[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:20 dist-test-slave-zpvx krb5kdc[15490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:20 dist-test-slave-zpvx krb5kdc[15490](info): set up 2 sockets
Aug 08 04:50:20 dist-test-slave-zpvx krb5kdc[15490](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 04:50:22.014508 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.433s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:22 dist-test-slave-zpvx krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:22.148525 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:37585
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:37585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:22.262831 15510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:22.263088 15510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:22.263128 15510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:22.266992 15510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:22.267048 15510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:22.267068 15510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:22.267086 15510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:22.267100 15510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:22.272187 15510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:37585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:22.273204 15510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:22.274161 15510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:22.283310 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:22.283313 15525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:22.283311 15520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:22.283365 15510 server_base.cc:1034] running on GCE node
I20260808 04:50:22.284157 15510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:22.285373 15510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:22.286566 15510 hybrid_clock.cc:648] HybridClock initialized: now 1786164622286551 us; error 103 us; skew 500 ppm
Aug 08 04:50:22 dist-test-slave-zpvx krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164622, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:22.289510 15510 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:22.290427 15510 webserver.cc:460] Webserver started at http://127.15.30.126:32837/ using document root <none> and password file <none>
I20260808 04:50:22.290988 15510 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:22.291082 15510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:22.291332 15510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:22.293603 15510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "07327e2c18aa4c3ea51c65132f316c6f"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "176faf744b2f920a771abca2a1114c8f"
server_key_iv: "870b34233aa1cd0a772a75387209bb7e"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.294205 15510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "07327e2c18aa4c3ea51c65132f316c6f"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "176faf744b2f920a771abca2a1114c8f"
server_key_iv: "870b34233aa1cd0a772a75387209bb7e"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.297911 15510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 04:50:22.300416 15535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:22.301637 15510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:22.301802 15510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "07327e2c18aa4c3ea51c65132f316c6f"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "176faf744b2f920a771abca2a1114c8f"
server_key_iv: "870b34233aa1cd0a772a75387209bb7e"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.301918 15510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:22.314072 15510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:22.317034 15510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:22.317231 15510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:22.324440 15510 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:37585
I20260808 04:50:22.324468 15635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:37585 every 8 connection(s)
I20260808 04:50:22.325331 15510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:22.328096 15636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:22.333839 15636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f: Bootstrap starting.
I20260808 04:50:22.335762 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15510
I20260808 04:50:22.335922 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:22.336637 15636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:22.337522 15636 log.cc:824] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:22.337596 15481 external_mini_cluster.cc:1428] Setting key 3d45855e6105b8205d3096888b3b66a5
I20260808 04:50:22.339735 15636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f: No bootstrap required, opened a new log
I20260808 04:50:22.343745 15636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07327e2c18aa4c3ea51c65132f316c6f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37585 } }
I20260808 04:50:22.343947 15636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:22.343981 15636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07327e2c18aa4c3ea51c65132f316c6f, State: Initialized, Role: FOLLOWER
I20260808 04:50:22.344471 15636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07327e2c18aa4c3ea51c65132f316c6f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37585 } }
I20260808 04:50:22.344597 15636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:22.344681 15636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:22.344792 15636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:22.345692 15636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07327e2c18aa4c3ea51c65132f316c6f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37585 } }
I20260808 04:50:22.346032 15636 leader_election.cc:302] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07327e2c18aa4c3ea51c65132f316c6f; no voters:
I20260808 04:50:22.346352 15636 leader_election.cc:288] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:22.346568 15643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:22.346841 15643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [term 1 LEADER]: Becoming Leader. State: Replica: 07327e2c18aa4c3ea51c65132f316c6f, State: Running, Role: LEADER
I20260808 04:50:22.347242 15643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07327e2c18aa4c3ea51c65132f316c6f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37585 } }
I20260808 04:50:22.347602 15636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:22.349114 15645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07327e2c18aa4c3ea51c65132f316c6f. Latest consensus state: current_term: 1 leader_uuid: "07327e2c18aa4c3ea51c65132f316c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07327e2c18aa4c3ea51c65132f316c6f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37585 } } }
I20260808 04:50:22.349248 15645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:22.349618 15644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07327e2c18aa4c3ea51c65132f316c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07327e2c18aa4c3ea51c65132f316c6f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37585 } } }
I20260808 04:50:22.349727 15644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:22.350028 15661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:22.352975 15661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 04:50:22 dist-test-slave-zpvx krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:22.358935 15661 catalog_manager.cc:1347] Generated new cluster ID: d87487fc76cd4add88acafbfd4cc934a
I20260808 04:50:22.358996 15661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:22.366751 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:22.340992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.15.30.126:37585)
0808 04:50:22.341481 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:22.341498 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:22.342748 (+ 1250us) server_negotiation.cc:407] Connection header received
0808 04:50:22.345761 (+ 3013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:22.345782 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:22.346148 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:22.346501 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:22.349840 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.351367 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.353136 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.353820 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.353990 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:22.358797 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:22.358804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:22.358819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:22.358853 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:22.362756 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:22.363478 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:22.363484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:22.363490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:22.363642 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:22.364023 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:22.364027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:22.364029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:22.364693 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:22.364916 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:22.365547 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 04:50:22.365845 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":161,"threads_started":1}
I20260808 04:50:22.389101 15661 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:22.390404 15661 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:22.401988 15661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07327e2c18aa4c3ea51c65132f316c6f: Generated new TSK 0
I20260808 04:50:22.402820 15661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:22.435696 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37585
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:22.572705 15677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:22.572955 15677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:22.572999 15677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:22.577059 15677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:22.577117 15677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:22.577201 15677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:22.582633 15677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:22.583643 15677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:22.584757 15677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:22.595085 15688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:22.595055 15685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:22.595307 15677 server_base.cc:1034] running on GCE node
W20260808 04:50:22.595486 15684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:22.595870 15677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:22.596545 15677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:22.597725 15677 hybrid_clock.cc:648] HybridClock initialized: now 1786164622597705 us; error 32 us; skew 500 ppm
Aug 08 04:50:22 dist-test-slave-zpvx krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164622, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:22.600874 15677 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:22.601742 15677 webserver.cc:460] Webserver started at http://127.15.30.65:37625/ using document root <none> and password file <none>
I20260808 04:50:22.602321 15677 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:22.602416 15677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:22.602651 15677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:22.604846 15677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "a852b30ef83946e495baec4227a912f7"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "aea12cbf6e49d6c023d0acce254c02be"
server_key_iv: "8d6a42de63d72c10d45d5c7857a45710"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.605451 15677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "a852b30ef83946e495baec4227a912f7"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "aea12cbf6e49d6c023d0acce254c02be"
server_key_iv: "8d6a42de63d72c10d45d5c7857a45710"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.609328 15677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:50:22.611807 15701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:22.613409 15677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:50:22.613533 15677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "a852b30ef83946e495baec4227a912f7"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "aea12cbf6e49d6c023d0acce254c02be"
server_key_iv: "8d6a42de63d72c10d45d5c7857a45710"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.613647 15677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:22.643826 15677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:22.647159 15677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:22.647334 15677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:22.648020 15677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:22.649048 15677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:22.649099 15677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:22.649175 15677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:22.649206 15677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:22.664052 15677 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:42665
I20260808 04:50:22.664108 15900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:42665 every 8 connection(s)
I20260808 04:50:22.665094 15677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:22.673491 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15677
I20260808 04:50:22.673578 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:22.673859 15481 external_mini_cluster.cc:1428] Setting key 848b06954463fcea09fa86e40f662894
Aug 08 04:50:22 dist-test-slave-zpvx krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164622, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:22.680115 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:22.667151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:39175 (local address 127.15.30.126:37585)
0808 04:50:22.667321 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:22.667325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:22.668646 (+ 1321us) server_negotiation.cc:407] Connection header received
0808 04:50:22.669872 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:22.669875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:22.669939 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:22.670042 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:22.671402 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.671947 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.673290 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.673518 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.673646 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:22.676470 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:22.676473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:22.676474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:22.676499 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:22.677938 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:22.678496 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:22.678500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:22.678502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:22.678637 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:22.678986 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:22.678990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:22.678992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:22.679175 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:22.679285 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:22.679826 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 04:50:22.679951 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:50:22.680969 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:22.667494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:37585 (local address 127.15.30.65:39175)
0808 04:50:22.668500 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:22.668545 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:22.669627 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:22.670227 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:22.670234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:22.670628 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:22.671249 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:22.671265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.672080 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.672084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:22.673154 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:22.673165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.673713 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.673717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:22.673888 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:22.674563 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:22.674583 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:22.676256 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:22.678079 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:22.678085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:22.678098 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:22.678363 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:22.678736 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:22.678740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:22.678742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:22.678886 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:22.679286 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:22.679294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:22.679584 (+ 290us) client_negotiation.cc:769] Sending connection context
0808 04:50:22.679782 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 04:50:22.680104 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":600,"threads_started":1}
I20260808 04:50:22.682423 15901 heartbeater.cc:347] Connected to a master server at 127.15.30.126:37585
I20260808 04:50:22.682657 15901 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:22.683143 15901 heartbeater.cc:511] Master 127.15.30.126:37585 requested a full tablet report, sending...
I20260808 04:50:22.684839 15567 ts_manager.cc:194] Registered new tserver with Master: a852b30ef83946e495baec4227a912f7 (127.15.30.65:42665)
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:50:22.686775 15567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:39175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:22.707262 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37585
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:22.843541 15908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:22.843797 15908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:22.843842 15908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:22.848294 15908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:22.848357 15908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:22.848531 15908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:22.854200 15908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:22.855227 15908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:22.856204 15908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:22.866153 15920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:22.866187 15927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:22.866176 15924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:22.866547 15908 server_base.cc:1034] running on GCE node
I20260808 04:50:22.866979 15908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:22.867614 15908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:22.868817 15908 hybrid_clock.cc:648] HybridClock initialized: now 1786164622868785 us; error 44 us; skew 500 ppm
Aug 08 04:50:22 dist-test-slave-zpvx krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164622, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:22.872766 15908 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:22.873719 15908 webserver.cc:460] Webserver started at http://127.15.30.66:36967/ using document root <none> and password file <none>
I20260808 04:50:22.874306 15908 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:22.874400 15908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:22.874617 15908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:22.876945 15908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "d82178818d4d49cc81b913b628b7393c"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "d1c192f7f1596b8747ab293d3550a365"
server_key_iv: "8caba688e9a0346d295dead5629545e2"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.877555 15908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "d82178818d4d49cc81b913b628b7393c"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "d1c192f7f1596b8747ab293d3550a365"
server_key_iv: "8caba688e9a0346d295dead5629545e2"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.881667 15908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:50:22.884441 15943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:22.885634 15908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:22.885792 15908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "d82178818d4d49cc81b913b628b7393c"
format_stamp: "Formatted at 2026-08-08 04:50:22 on dist-test-slave-zpvx"
server_key: "d1c192f7f1596b8747ab293d3550a365"
server_key_iv: "8caba688e9a0346d295dead5629545e2"
server_key_version: "encryptionkey@0"
I20260808 04:50:22.885905 15908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:22.910406 15908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:22.913821 15908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:22.913978 15908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:22.914505 15908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:22.915303 15908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:22.915351 15908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:22.915392 15908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:22.915414 15908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:22.927645 15908 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:34397
I20260808 04:50:22.927700 16141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:34397 every 8 connection(s)
I20260808 04:50:22.928639 15908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:22.934513 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15908
I20260808 04:50:22.934635 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:22.934890 15481 external_mini_cluster.cc:1428] Setting key fbebb8dddb7341ad6d8103171f7a894f
Aug 08 04:50:22 dist-test-slave-zpvx krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164622, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:22.944576 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:22.930572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:44299 (local address 127.15.30.126:37585)
0808 04:50:22.930726 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:22.930730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:22.931859 (+ 1129us) server_negotiation.cc:407] Connection header received
0808 04:50:22.932890 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:22.932893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:22.932941 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:22.933017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:22.934432 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.935091 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.936878 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.937111 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.937195 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:22.940836 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:22.940841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:22.940845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:22.940881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:22.942411 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:22.942951 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:22.942960 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:22.942961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:22.943039 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:22.943399 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:22.943402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:22.943404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:22.943591 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:22.943702 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:22.944256 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 04:50:22.944410 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:50:22.945466 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:22.930928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:37585 (local address 127.15.30.66:44299)
0808 04:50:22.931710 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:22.931753 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:22.932700 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:22.933188 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:22.933194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:22.933699 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:22.934268 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:22.934295 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.935224 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.935228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:22.936747 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:22.936758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:22.937276 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:22.937281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:22.937517 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:22.938215 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:22.938241 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:22.940648 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:22.942537 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:22.942543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:22.942559 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:22.942837 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:22.943134 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:22.943137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:22.943139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:22.943291 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:22.943701 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:22.943708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:22.944002 (+ 294us) client_negotiation.cc:769] Sending connection context
0808 04:50:22.944198 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 04:50:22.944574 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":140,"threads_started":1}
I20260808 04:50:22.946730 16143 heartbeater.cc:347] Connected to a master server at 127.15.30.126:37585
I20260808 04:50:22.947113 16143 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:50:22.947758 16143 heartbeater.cc:511] Master 127.15.30.126:37585 requested a full tablet report, sending...
I20260808 04:50:22.948954 15567 ts_manager.cc:194] Registered new tserver with Master: d82178818d4d49cc81b913b628b7393c (127.15.30.66:34397)
I20260808 04:50:22.949671 15567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:44299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:22.966881 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37585
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:23.101181 16152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:23.101433 16152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:23.101477 16152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:23.105530 16152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:23.105628 16152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:23.105736 16152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:23.111092 16152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:23.112149 16152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:23.113240 16152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:23.123445 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:23.123493 16161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:23.123509 16160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:23.124025 16152 server_base.cc:1034] running on GCE node
I20260808 04:50:23.124500 16152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:23.125144 16152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:23.126314 16152 hybrid_clock.cc:648] HybridClock initialized: now 1786164623126288 us; error 36 us; skew 500 ppm
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164623, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:23.129158 16152 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:23.130059 16152 webserver.cc:460] Webserver started at http://127.15.30.67:43285/ using document root <none> and password file <none>
I20260808 04:50:23.130630 16152 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:23.130735 16152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:23.130977 16152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:23.133147 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "89fc227906dc47dd9f319e7d7b5f1855"
format_stamp: "Formatted at 2026-08-08 04:50:23 on dist-test-slave-zpvx"
server_key: "fd0857c9f7c63ce2de1339aaa9656e1f"
server_key_iv: "6a4152b037c383c4e2e3de1d5eecec21"
server_key_version: "encryptionkey@0"
I20260808 04:50:23.133785 16152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "89fc227906dc47dd9f319e7d7b5f1855"
format_stamp: "Formatted at 2026-08-08 04:50:23 on dist-test-slave-zpvx"
server_key: "fd0857c9f7c63ce2de1339aaa9656e1f"
server_key_iv: "6a4152b037c383c4e2e3de1d5eecec21"
server_key_version: "encryptionkey@0"
I20260808 04:50:23.137792 16152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 04:50:23.140271 16176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:23.141530 16152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:23.141680 16152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "89fc227906dc47dd9f319e7d7b5f1855"
format_stamp: "Formatted at 2026-08-08 04:50:23 on dist-test-slave-zpvx"
server_key: "fd0857c9f7c63ce2de1339aaa9656e1f"
server_key_iv: "6a4152b037c383c4e2e3de1d5eecec21"
server_key_version: "encryptionkey@0"
I20260808 04:50:23.141798 16152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:23.178733 16152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:23.182004 16152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:23.182277 16152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:23.182981 16152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:23.183852 16152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:23.183928 16152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:23.184000 16152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:23.184053 16152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:23.198930 16152 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:43767
I20260808 04:50:23.198980 16374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:43767 every 8 connection(s)
I20260808 04:50:23.199956 16152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:23.203634 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16152
I20260808 04:50:23.203785 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:23.204139 15481 external_mini_cluster.cc:1428] Setting key d7227de3ddec16c8f4391380834f4435
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164623, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:23.215180 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:23.202067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:56953 (local address 127.15.30.126:37585)
0808 04:50:23.202208 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:23.202212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:23.203274 (+ 1062us) server_negotiation.cc:407] Connection header received
0808 04:50:23.204839 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:23.204843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:23.204911 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:23.205010 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:23.206822 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.207357 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.208634 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.208869 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.208977 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:23.211597 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:23.211603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:23.211612 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:23.211637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:23.213118 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:23.213649 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:23.213652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:23.213654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:23.213732 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:23.214067 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:23.214072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:23.214074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:23.214264 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:23.214392 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:23.214877 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:50:23.215007 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:50:23.215978 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:23.202391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:37585 (local address 127.15.30.67:56953)
0808 04:50:23.203109 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:23.203149 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:23.204546 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:23.205210 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:23.205221 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:23.205902 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:23.206665 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:23.206685 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.207486 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.207489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:23.208509 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:23.208519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.209041 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.209046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:23.209229 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:23.209844 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:23.209862 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:23.211437 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:23.213243 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:23.213249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:23.213265 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:23.213526 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:23.213825 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:23.213829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:23.213831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:23.213970 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:23.214390 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:23.214397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:23.214636 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 04:50:23.214833 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 04:50:23.215146 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":138,"threads_started":1}
I20260808 04:50:23.217162 16375 heartbeater.cc:347] Connected to a master server at 127.15.30.126:37585
I20260808 04:50:23.217417 16375 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:23.218044 16375 heartbeater.cc:511] Master 127.15.30.126:37585 requested a full tablet report, sending...
I20260808 04:50:23.219139 15567 ts_manager.cc:194] Registered new tserver with Master: 89fc227906dc47dd9f319e7d7b5f1855 (127.15.30.67:43767)
I20260808 04:50:23.219723 15567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:56953
I20260808 04:50:23.230348 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:23.258671 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:23.249334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.15.30.65:42665)
0808 04:50:23.249684 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:23.249691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:23.249717 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:50:23.249841 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:23.249846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:23.250050 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:23.250178 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:23.250911 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.251689 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.252576 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.252943 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.253028 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:23.255421 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:23.255429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:23.255441 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:23.255465 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:23.256993 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:23.257397 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:23.257402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:23.257403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:23.257482 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:23.257730 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:23.257733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:23.257734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:23.257954 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:23.258099 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:23.258330 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 04:50:23.258497 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
W20260808 04:50:23.259619 15749 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60998
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:23.286224 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:23.278047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.15.30.65:42665)
0808 04:50:23.278256 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:23.278268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:23.278572 (+ 304us) server_negotiation.cc:407] Connection header received
0808 04:50:23.278726 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:23.278729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:23.278779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:23.278856 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:23.279522 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.280012 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.280961 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:23.281175 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:23.281262 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:23.283466 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:23.283469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:23.283471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:23.283495 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:23.284805 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:23.285170 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:23.285172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:23.285173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:23.285257 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:23.285482 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:23.285484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:23.285485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:23.285630 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:23.285728 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:23.285914 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 04:50:23.286007 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 04:50:23.287287 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15677
I20260808 04:50:23.301723 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15908
I20260808 04:50:23.313781 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16152
I20260808 04:50:23.327349 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15510
2026-08-08T04:50:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2782 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[16392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[16392](info): set up 2 sockets
Aug 08 04:50:23 dist-test-slave-zpvx krb5kdc[16392](info): commencing operation
krb5kdc: starting...
W20260808 04:50:24.576910 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:24 dist-test-slave-zpvx krb5kdc[16392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:24.704097 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:32911
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:24.814939 16421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:24.815177 16421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:24.815215 16421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:24.819098 16421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:24.819149 16421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:24.819168 16421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:24.819185 16421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:24.819198 16421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:24.824084 16421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:32911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:24.825031 16421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:24.825918 16421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:24.834226 16429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:24.834254 16427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:24.834241 16431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:24.834524 16421 server_base.cc:1034] running on GCE node
I20260808 04:50:24.835104 16421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:24.836202 16421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:24.837395 16421 hybrid_clock.cc:648] HybridClock initialized: now 1786164624837374 us; error 32 us; skew 500 ppm
Aug 08 04:50:24 dist-test-slave-zpvx krb5kdc[16392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164624, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:24.840288 16421 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:24.841161 16421 webserver.cc:460] Webserver started at http://127.15.30.126:32807/ using document root <none> and password file <none>
I20260808 04:50:24.841725 16421 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:24.841818 16421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:24.842047 16421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:24.844046 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "16d35c888a5c40bd97b7e7fbce51f1a2"
format_stamp: "Formatted at 2026-08-08 04:50:24 on dist-test-slave-zpvx"
server_key: "d28feaa66cc6f3212988fbd7b799260a"
server_key_iv: "1709841d7f4dce19a2d11de6b3366738"
server_key_version: "encryptionkey@0"
I20260808 04:50:24.844635 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "16d35c888a5c40bd97b7e7fbce51f1a2"
format_stamp: "Formatted at 2026-08-08 04:50:24 on dist-test-slave-zpvx"
server_key: "d28feaa66cc6f3212988fbd7b799260a"
server_key_iv: "1709841d7f4dce19a2d11de6b3366738"
server_key_version: "encryptionkey@0"
I20260808 04:50:24.848250 16421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:50:24.850600 16445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:24.851727 16421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:24.851876 16421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "16d35c888a5c40bd97b7e7fbce51f1a2"
format_stamp: "Formatted at 2026-08-08 04:50:24 on dist-test-slave-zpvx"
server_key: "d28feaa66cc6f3212988fbd7b799260a"
server_key_iv: "1709841d7f4dce19a2d11de6b3366738"
server_key_version: "encryptionkey@0"
I20260808 04:50:24.851984 16421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:24.870242 16421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:24.873683 16421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:24.873891 16421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:24.880928 16421 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:32911
I20260808 04:50:24.880993 16534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:32911 every 8 connection(s)
I20260808 04:50:24.881880 16421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:24.884603 16536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:24.889835 16536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2: Bootstrap starting.
I20260808 04:50:24.889967 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16421
I20260808 04:50:24.890102 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:24.890383 15481 external_mini_cluster.cc:1428] Setting key f8a5c08c46ecd90b03a2d1fd9db30c20
I20260808 04:50:24.892467 16536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:24.893189 16536 log.cc:824] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:24.894969 16536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2: No bootstrap required, opened a new log
I20260808 04:50:24.898499 16536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 32911 } }
I20260808 04:50:24.898801 16536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:24.898872 16536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d35c888a5c40bd97b7e7fbce51f1a2, State: Initialized, Role: FOLLOWER
Aug 08 04:50:24 dist-test-slave-zpvx krb5kdc[16392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:24.899362 16536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 32911 } }
I20260808 04:50:24.899542 16536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:24.899616 16536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:24.899724 16536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:24.900664 16536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 32911 } }
I20260808 04:50:24.900995 16536 leader_election.cc:302] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d35c888a5c40bd97b7e7fbce51f1a2; no voters:
I20260808 04:50:24.901314 16536 leader_election.cc:288] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:24.901386 16543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:24.901597 16543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [term 1 LEADER]: Becoming Leader. State: Replica: 16d35c888a5c40bd97b7e7fbce51f1a2, State: Running, Role: LEADER
I20260808 04:50:24.901929 16543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 32911 } }
I20260808 04:50:24.902551 16536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:24.903074 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d35c888a5c40bd97b7e7fbce51f1a2. Latest consensus state: current_term: 1 leader_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 32911 } } }
I20260808 04:50:24.903309 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:24.903523 16546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d35c888a5c40bd97b7e7fbce51f1a2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 32911 } } }
I20260808 04:50:24.903716 16546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:24.903772 16561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:24.905493 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:24.891673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.15.30.126:32911)
0808 04:50:24.892095 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:24.892114 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:24.892148 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:50:24.892907 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:24.892927 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:24.893280 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:24.893632 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:24.894431 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:24.895453 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:24.896309 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:24.896837 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:24.897019 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:24.899882 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:24.899889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:24.899906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:24.899938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:24.902009 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:24.902490 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:24.902494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:24.902498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:24.902605 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:24.902982 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:24.902986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:24.902988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:24.903536 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:24.903732 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:24.904186 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 04:50:24.904536 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260808 04:50:24.907148 16561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:24.912464 16561 catalog_manager.cc:1347] Generated new cluster ID: 58601268cc474e3ab604f8ec26f3a7e3
I20260808 04:50:24.912528 16561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:24.931458 16561 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:24.932613 16561 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:24.943915 16561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d35c888a5c40bd97b7e7fbce51f1a2: Generated new TSK 0
I20260808 04:50:24.944574 16561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:24.977384 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:32911
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:25.098740 16584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:25.099027 16584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:25.099107 16584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:25.103308 16584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:25.103384 16584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:25.103473 16584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:25.108819 16584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:32911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:25.109915 16584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:25.110984 16584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:25.120824 16591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:25.120811 16593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:25.120811 16590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:25.121133 16584 server_base.cc:1034] running on GCE node
I20260808 04:50:25.121491 16584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:25.122129 16584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:25.123294 16584 hybrid_clock.cc:648] HybridClock initialized: now 1786164625123276 us; error 34 us; skew 500 ppm
Aug 08 04:50:25 dist-test-slave-zpvx krb5kdc[16392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164625, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:25.126377 16584 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:25.127281 16584 webserver.cc:460] Webserver started at http://127.15.30.65:42311/ using document root <none> and password file <none>
I20260808 04:50:25.127847 16584 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:25.127912 16584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:25.128162 16584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:25.130286 16584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "a2e5e111a3a84d61af084fd50cbfddad"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "d90246b067259aac9371ac71d04e26bc"
server_key_iv: "eeb053ae982b42fcd84b16cdb4c6ba22"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.130851 16584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "a2e5e111a3a84d61af084fd50cbfddad"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "d90246b067259aac9371ac71d04e26bc"
server_key_iv: "eeb053ae982b42fcd84b16cdb4c6ba22"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.134526 16584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260808 04:50:25.136893 16606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.138033 16584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:25.138154 16584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "a2e5e111a3a84d61af084fd50cbfddad"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "d90246b067259aac9371ac71d04e26bc"
server_key_iv: "eeb053ae982b42fcd84b16cdb4c6ba22"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.138255 16584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:25.165511 16584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:25.168494 16584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:25.168700 16584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:25.169327 16584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:25.170159 16584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:25.170210 16584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.170277 16584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:25.170310 16584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.179344 16584 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:39875
I20260808 04:50:25.179378 16815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:39875 every 8 connection(s)
I20260808 04:50:25.180231 16584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:25.184618 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16584
I20260808 04:50:25.184736 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:25.184999 15481 external_mini_cluster.cc:1428] Setting key f3286c9a4d0fb086b95b865bfa640c96
Aug 08 04:50:25 dist-test-slave-zpvx krb5kdc[16392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164625, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:25.194666 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.182040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:56753 (local address 127.15.30.126:32911)
0808 04:50:25.182202 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:25.182206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:25.183009 (+ 803us) server_negotiation.cc:407] Connection header received
0808 04:50:25.184325 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:25.184328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:25.184414 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:25.184525 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:25.185918 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.186549 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.187686 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.187945 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.188056 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.190631 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:25.190634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:25.190636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:25.190659 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:25.192471 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.192975 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.192981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.192985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.193093 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.193443 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.193448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.193452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.193736 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:25.193851 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:25.194312 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 04:50:25.194464 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:50:25.195346 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.182412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:32911 (local address 127.15.30.65:56753)
0808 04:50:25.182862 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:25.182906 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:25.184115 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:25.184731 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:25.184742 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:25.185193 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:25.185771 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.185786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.186693 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.186696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.187542 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.187551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.188108 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.188113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.188279 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.188933 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:25.188949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:25.190461 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:25.192591 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:25.192596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:25.192607 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:25.192866 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:25.193202 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:25.193204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:25.193206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:25.193338 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:25.193845 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:25.193850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:25.194062 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 04:50:25.194217 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 04:50:25.194456 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:50:25.196708 16817 heartbeater.cc:347] Connected to a master server at 127.15.30.126:32911
I20260808 04:50:25.196960 16817 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:25.197621 16817 heartbeater.cc:511] Master 127.15.30.126:32911 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:50:25.199288 16471 ts_manager.cc:194] Registered new tserver with Master: a2e5e111a3a84d61af084fd50cbfddad (127.15.30.65:39875)
I20260808 04:50:25.200543 16471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:56753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:25.215039 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:32911
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:25.326375 16825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:25.326606 16825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:25.326673 16825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:25.330586 16825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:25.330636 16825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:25.330736 16825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:25.335832 16825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:32911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:25.336810 16825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:25.337790 16825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:25.347241 16832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:25.347393 16825 server_base.cc:1034] running on GCE node
W20260808 04:50:25.347241 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:25.347266 16837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:25.347939 16825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:25.348620 16825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:25.349824 16825 hybrid_clock.cc:648] HybridClock initialized: now 1786164625349809 us; error 27 us; skew 500 ppm
Aug 08 04:50:25 dist-test-slave-zpvx krb5kdc[16392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164625, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:25.352659 16825 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:25.353480 16825 webserver.cc:460] Webserver started at http://127.15.30.66:38369/ using document root <none> and password file <none>
I20260808 04:50:25.354113 16825 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:25.354213 16825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:25.354452 16825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:25.356805 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "b62f619ec94747cea7280ee86234f69a"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "1a3851b76aab85775def057b3fdb915d"
server_key_iv: "a0fc0d459be13461aa13b0d7f93f66dd"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.357394 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "b62f619ec94747cea7280ee86234f69a"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "1a3851b76aab85775def057b3fdb915d"
server_key_iv: "a0fc0d459be13461aa13b0d7f93f66dd"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.361071 16825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:50:25.363410 16851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.364800 16825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:25.364928 16825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "b62f619ec94747cea7280ee86234f69a"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "1a3851b76aab85775def057b3fdb915d"
server_key_iv: "a0fc0d459be13461aa13b0d7f93f66dd"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.365041 16825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:25.379611 16825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:25.382812 16825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:25.383008 16825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:25.383567 16825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:25.384357 16825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:25.384450 16825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.384521 16825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:25.384563 16825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.393198 16825 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:32835
I20260808 04:50:25.393237 17043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:32835 every 8 connection(s)
I20260808 04:50:25.394091 16825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:25.401187 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16825
I20260808 04:50:25.401322 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:25.401623 15481 external_mini_cluster.cc:1428] Setting key 30127b9d4081af5d77c52f5115f1bb77
Aug 08 04:50:25 dist-test-slave-zpvx krb5kdc[16392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164625, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:25.407974 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.395805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:47875 (local address 127.15.30.126:32911)
0808 04:50:25.395951 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:25.395956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:25.396821 (+ 865us) server_negotiation.cc:407] Connection header received
0808 04:50:25.398114 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:25.398117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:25.398164 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:25.398240 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:25.399322 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.399839 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.401014 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.401365 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.401481 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.404221 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:25.404228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:25.404234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:25.404266 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:25.405945 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.406452 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.406457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.406459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.406542 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.406890 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.406893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.406896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.407073 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:25.407183 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:25.407681 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 04:50:25.407809 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:50:25.408857 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.396191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:32911 (local address 127.15.30.66:47875)
0808 04:50:25.396675 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:25.396716 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:25.397899 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:25.398407 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:25.398414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:25.398745 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:25.399188 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.399200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.399946 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.399949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.400886 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.400896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.401506 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.401509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.401648 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.402346 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:25.402368 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:25.404061 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:25.406065 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:25.406071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:25.406085 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:25.406344 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:25.406639 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:25.406642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:25.406644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:25.406783 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:25.407181 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:25.407188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:25.407424 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 04:50:25.407626 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 04:50:25.407925 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":159,"threads_started":1}
I20260808 04:50:25.410037 17044 heartbeater.cc:347] Connected to a master server at 127.15.30.126:32911
I20260808 04:50:25.410292 17044 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:25.410761 17044 heartbeater.cc:511] Master 127.15.30.126:32911 requested a full tablet report, sending...
I20260808 04:50:25.411789 16471 ts_manager.cc:194] Registered new tserver with Master: b62f619ec94747cea7280ee86234f69a (127.15.30.66:32835)
I20260808 04:50:25.412335 16471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:47875
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:25.431473 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:32911
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:25.543380 17050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:25.543632 17050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:25.543701 17050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:25.547616 17050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:25.547696 17050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:25.547785 17050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:25.552836 17050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:32911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:25.553798 17050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:25.554735 17050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:25.564177 17065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:25.564177 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:25.564356 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:25.564447 17050 server_base.cc:1034] running on GCE node
I20260808 04:50:25.564916 17050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:25.565531 17050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:25.566727 17050 hybrid_clock.cc:648] HybridClock initialized: now 1786164625566704 us; error 40 us; skew 500 ppm
Aug 08 04:50:25 dist-test-slave-zpvx krb5kdc[16392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164625, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:25.569588 17050 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:25.570456 17050 webserver.cc:460] Webserver started at http://127.15.30.67:37683/ using document root <none> and password file <none>
I20260808 04:50:25.571007 17050 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:25.571096 17050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:25.571328 17050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:25.573360 17050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "79cc3bdbde9945409456138086a4759a"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "d0f37d342bd5e7a716c5e3474fbd245b"
server_key_iv: "f0e40c5dc4101a18f4524d0c1d530cde"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.573931 17050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "79cc3bdbde9945409456138086a4759a"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "d0f37d342bd5e7a716c5e3474fbd245b"
server_key_iv: "f0e40c5dc4101a18f4524d0c1d530cde"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.577636 17050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:50:25.579977 17075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.581159 17050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:25.581295 17050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "79cc3bdbde9945409456138086a4759a"
format_stamp: "Formatted at 2026-08-08 04:50:25 on dist-test-slave-zpvx"
server_key: "d0f37d342bd5e7a716c5e3474fbd245b"
server_key_iv: "f0e40c5dc4101a18f4524d0c1d530cde"
server_key_version: "encryptionkey@0"
I20260808 04:50:25.581403 17050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:25.606863 17050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:25.610128 17050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:25.610332 17050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:25.610921 17050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:25.611724 17050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:25.611792 17050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.611864 17050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:25.611914 17050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:25.621821 17050 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:45085
I20260808 04:50:25.621845 17258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:45085 every 8 connection(s)
I20260808 04:50:25.622696 17050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:25.627965 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17050
I20260808 04:50:25.628090 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:25.628424 15481 external_mini_cluster.cc:1428] Setting key fad9571e01ffcd8d3cefc96d65970e71
Aug 08 04:50:25 dist-test-slave-zpvx krb5kdc[16392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164625, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:25.641716 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.624552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:34533 (local address 127.15.30.126:32911)
0808 04:50:25.624685 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:25.624689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:25.625503 (+ 814us) server_negotiation.cc:407] Connection header received
0808 04:50:25.626931 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:25.626941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:25.627020 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:25.627117 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:25.629179 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.629926 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.631402 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.631641 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.631761 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.638274 (+ 6513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:25.638277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:25.638279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:25.638303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:25.639732 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.640166 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.640169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.640170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.640245 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.640598 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.640601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.640602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.640784 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:25.640859 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:25.641404 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 04:50:25.641521 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:50:25.642571 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.624871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:32911 (local address 127.15.30.67:34533)
0808 04:50:25.625349 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:25.625390 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:25.626688 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:25.627360 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:25.627368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:25.627885 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:25.628741 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.628763 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.630035 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.630038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.631272 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.631282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.631792 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.631796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.631975 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.636659 (+ 4684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:25.636676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:25.638119 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:25.639828 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:25.639832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:25.639843 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:25.640075 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:25.640321 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:25.640323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:25.640325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:25.640507 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:25.640872 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:25.640877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:25.641130 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 04:50:25.641327 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 04:50:25.641637 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":159,"threads_started":1}
I20260808 04:50:25.643639 17261 heartbeater.cc:347] Connected to a master server at 127.15.30.126:32911
I20260808 04:50:25.643857 17261 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:25.644291 17261 heartbeater.cc:511] Master 127.15.30.126:32911 requested a full tablet report, sending...
I20260808 04:50:25.645273 16471 ts_manager.cc:194] Registered new tserver with Master: 79cc3bdbde9945409456138086a4759a (127.15.30.67:45085)
I20260808 04:50:25.645818 16471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:34533
I20260808 04:50:25.654124 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:50:25.666162 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.658027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.15.30.126:32911)
0808 04:50:25.658151 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:25.658178 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:25.658433 (+ 255us) server_negotiation.cc:407] Connection header received
0808 04:50:25.658643 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:25.658646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:25.658689 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:25.658781 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:25.659565 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.660127 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.661109 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.661349 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.661473 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.662882 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:25.662887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:25.662890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:25.662924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:25.664215 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.664728 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.664733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.664735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.664847 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:25.665160 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:25.665163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:25.665165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:25.665404 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:25.665566 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:25.665838 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 04:50:25.665970 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:50:25.670881 16471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:25.673974 16471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:25.688362 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.681979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:32835 (local address 127.0.0.1:50320)
0808 04:50:25.682506 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:25.682554 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:25.682692 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:25.683800 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:25.683803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:25.683822 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:25.684207 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.684213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.685473 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.685479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.686652 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.686661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.687788 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.687792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.687897 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.687935 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:50:25.688073 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 04:50:25.688188 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":30,"spinlock_wait_cycles":5120,"thread_start_us":82,"threads_started":1}
I20260808 04:50:25.688418 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.682682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:39875 (local address 127.0.0.1:57302)
0808 04:50:25.682907 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:25.682921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:25.683004 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:25.683930 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:25.683932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:25.683949 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:25.684058 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.684066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.685920 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.685923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.687198 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.687208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.687992 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.687996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.688078 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.688100 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:50:25.688184 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:50:25.688262 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":128,"thread_start_us":99,"threads_started":1}
I20260808 04:50:25.688474 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.683097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.15.30.65:39875)
0808 04:50:25.683505 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:25.683520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:25.683539 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:50:25.683607 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:25.683612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:25.683785 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:25.684008 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:25.684182 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.685577 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.687325 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.687864 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.687980 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.688110 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:25.688216 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 04:50:25.688301 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":130,"threads_started":1}
I20260808 04:50:25.688776 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.682060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:45085 (local address 127.0.0.1:49912)
0808 04:50:25.682506 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:25.682553 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:25.682691 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:25.683759 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:25.683767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:25.683789 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:25.684058 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.684066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.685903 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.685908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.687238 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:25.687250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.688412 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.688416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:25.688513 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.688539 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:50:25.688592 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 04:50:25.688651 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":139,"spinlock_wait_cycles":6272,"thread_start_us":88,"threads_started":1}
I20260808 04:50:25.688817 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.682097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.15.30.66:32835)
0808 04:50:25.682983 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:25.682996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:25.683019 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:50:25.683095 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:25.683100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:25.683323 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:25.683537 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:25.684323 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.685349 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.686776 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.687485 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.687578 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.687725 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:25.688561 (+ 836us) server_negotiation.cc:299] Negotiation successful
0808 04:50:25.688654 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":113,"threads_started":1}
I20260808 04:50:25.690074 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:25.682780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.15.30.67:45085)
0808 04:50:25.683279 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:25.683291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:25.683313 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:50:25.683390 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:25.683396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:25.683576 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:25.683902 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:25.684202 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.685635 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.687381 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:25.688250 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:25.689686 (+ 1436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:25.689816 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:25.689854 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 04:50:25.689913 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":107,"threads_started":1}
I20260808 04:50:25.691828 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 2031671f516345e7a6545df890b845a3 (DEFAULT_TABLE table=test-table [id=7a26c0d88ded41bea67e31971aabdc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:25.691828 16935 tablet_service.cc:1467] Processing CreateTablet for tablet 2031671f516345e7a6545df890b845a3 (DEFAULT_TABLE table=test-table [id=7a26c0d88ded41bea67e31971aabdc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:25.692327 17157 tablet_service.cc:1467] Processing CreateTablet for tablet 2031671f516345e7a6545df890b845a3 (DEFAULT_TABLE table=test-table [id=7a26c0d88ded41bea67e31971aabdc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:25.693053 16935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2031671f516345e7a6545df890b845a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:25.693053 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2031671f516345e7a6545df890b845a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:25.693535 17157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2031671f516345e7a6545df890b845a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:25.700960 17290 tablet_bootstrap.cc:492] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a: Bootstrap starting.
I20260808 04:50:25.701342 17292 tablet_bootstrap.cc:492] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad: Bootstrap starting.
I20260808 04:50:25.702288 17293 tablet_bootstrap.cc:492] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a: Bootstrap starting.
I20260808 04:50:25.704017 17292 tablet_bootstrap.cc:654] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:25.704017 17290 tablet_bootstrap.cc:654] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:25.705435 17293 tablet_bootstrap.cc:654] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:25.705471 17292 log.cc:824] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:25.705883 17290 log.cc:824] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:25.706405 17293 log.cc:824] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:25.707919 17290 tablet_bootstrap.cc:492] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a: No bootstrap required, opened a new log
I20260808 04:50:25.708115 17290 ts_tablet_manager.cc:1397] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:50:25.708248 17293 tablet_bootstrap.cc:492] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a: No bootstrap required, opened a new log
I20260808 04:50:25.708483 17293 ts_tablet_manager.cc:1397] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:50:25.708513 17292 tablet_bootstrap.cc:492] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad: No bootstrap required, opened a new log
I20260808 04:50:25.708675 17292 ts_tablet_manager.cc:1397] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260808 04:50:25.712026 17293 raft_consensus.cc:348] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:25.712270 17293 raft_consensus.cc:374] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:25.712347 17293 raft_consensus.cc:729] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62f619ec94747cea7280ee86234f69a, State: Initialized, Role: FOLLOWER
I20260808 04:50:25.712819 17293 consensus_queue.cc:260] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:25.713135 17290 raft_consensus.cc:348] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:25.713459 17290 raft_consensus.cc:374] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:25.713527 17290 raft_consensus.cc:729] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc3bdbde9945409456138086a4759a, State: Initialized, Role: FOLLOWER
I20260808 04:50:25.713572 17044 heartbeater.cc:503] Master 127.15.30.126:32911 was elected leader, sending a full tablet report...
I20260808 04:50:25.713727 17292 raft_consensus.cc:348] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:25.713963 17292 raft_consensus.cc:374] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:25.713967 17290 consensus_queue.cc:260] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:25.714056 17292 raft_consensus.cc:729] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e5e111a3a84d61af084fd50cbfddad, State: Initialized, Role: FOLLOWER
I20260808 04:50:25.714484 17292 consensus_queue.cc:260] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:25.714973 17293 ts_tablet_manager.cc:1428] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:50:25.714983 17290 ts_tablet_manager.cc:1428] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260808 04:50:25.715301 17261 heartbeater.cc:503] Master 127.15.30.126:32911 was elected leader, sending a full tablet report...
I20260808 04:50:25.715425 17292 ts_tablet_manager.cc:1428] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260808 04:50:25.715752 16817 heartbeater.cc:503] Master 127.15.30.126:32911 was elected leader, sending a full tablet report...
W20260808 04:50:25.873750 17262 tablet.cc:2369] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:50:25.895468 17045 tablet.cc:2369] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:50:25.931672 16819 tablet.cc:2369] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:26.043859 17301 raft_consensus.cc:482] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:26.044112 17301 raft_consensus.cc:504] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:26.045388 17301 leader_election.cc:288] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79cc3bdbde9945409456138086a4759a (127.15.30.67:45085), a2e5e111a3a84d61af084fd50cbfddad (127.15.30.65:39875)
I20260808 04:50:26.051255 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:26.045932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:58367 (local address 127.15.30.65:39875)
0808 04:50:26.046062 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:26.046065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:26.046230 (+ 165us) server_negotiation.cc:407] Connection header received
0808 04:50:26.046310 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:26.046315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:26.046365 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:26.046437 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:26.046789 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.047614 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.049783 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.050472 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.050601 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:26.050652 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:26.051055 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 04:50:26.051125 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:50:26.051270 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:26.045766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:44165 (local address 127.15.30.67:45085)
0808 04:50:26.045954 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:26.045960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:26.045978 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:50:26.046157 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:26.046160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:26.046236 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:26.046335 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:26.046736 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.047614 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.049782 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.050480 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.050602 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:26.050652 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:26.051055 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 04:50:26.051124 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:50:26.051334 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:26.045540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:45085 (local address 127.15.30.66:44165)
0808 04:50:26.045889 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:26.045906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:26.046022 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:26.046373 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:26.046375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:26.046392 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:26.046569 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:26.046577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.047757 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.047761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:26.049626 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:26.049636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.050646 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.050650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:26.050807 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:26.050917 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 04:50:26.051019 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:50:26.051121 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":121,"thread_start_us":75,"threads_started":1}
I20260808 04:50:26.051357 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:26.045692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:39875 (local address 127.15.30.66:58367)
0808 04:50:26.046120 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:26.046137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:26.046233 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:26.046467 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:26.046470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:26.046484 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:26.046659 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:26.046669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.047757 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.047761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:26.049623 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:26.049636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.050646 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.050650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:26.050930 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:26.050950 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 04:50:26.051023 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 04:50:26.051146 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":256,"thread_start_us":105,"threads_started":1}
I20260808 04:50:26.052152 16732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2031671f516345e7a6545df890b845a3" candidate_uuid: "b62f619ec94747cea7280ee86234f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e5e111a3a84d61af084fd50cbfddad" is_pre_election: true
I20260808 04:50:26.052150 17185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2031671f516345e7a6545df890b845a3" candidate_uuid: "b62f619ec94747cea7280ee86234f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cc3bdbde9945409456138086a4759a" is_pre_election: true
I20260808 04:50:26.052502 16732 raft_consensus.cc:2454] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b62f619ec94747cea7280ee86234f69a in term 0.
I20260808 04:50:26.052517 17185 raft_consensus.cc:2454] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b62f619ec94747cea7280ee86234f69a in term 0.
I20260808 04:50:26.053036 16855 leader_election.cc:302] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e5e111a3a84d61af084fd50cbfddad, b62f619ec94747cea7280ee86234f69a; no voters:
I20260808 04:50:26.053287 17301 raft_consensus.cc:2781] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:26.053375 17301 raft_consensus.cc:482] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:26.053416 17301 raft_consensus.cc:3037] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:26.054734 17301 raft_consensus.cc:504] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:26.055198 17301 leader_election.cc:288] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [CANDIDATE]: Term 1 election: Requested vote from peers 79cc3bdbde9945409456138086a4759a (127.15.30.67:45085), a2e5e111a3a84d61af084fd50cbfddad (127.15.30.65:39875)
I20260808 04:50:26.055441 17185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2031671f516345e7a6545df890b845a3" candidate_uuid: "b62f619ec94747cea7280ee86234f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cc3bdbde9945409456138086a4759a"
I20260808 04:50:26.055485 16732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2031671f516345e7a6545df890b845a3" candidate_uuid: "b62f619ec94747cea7280ee86234f69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e5e111a3a84d61af084fd50cbfddad"
I20260808 04:50:26.055601 17185 raft_consensus.cc:3037] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:26.055658 16732 raft_consensus.cc:3037] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:26.057063 16732 raft_consensus.cc:2454] T 2031671f516345e7a6545df890b845a3 P a2e5e111a3a84d61af084fd50cbfddad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b62f619ec94747cea7280ee86234f69a in term 1.
I20260808 04:50:26.057138 17185 raft_consensus.cc:2454] T 2031671f516345e7a6545df890b845a3 P 79cc3bdbde9945409456138086a4759a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b62f619ec94747cea7280ee86234f69a in term 1.
I20260808 04:50:26.057390 16855 leader_election.cc:302] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e5e111a3a84d61af084fd50cbfddad, b62f619ec94747cea7280ee86234f69a; no voters:
I20260808 04:50:26.057606 17301 raft_consensus.cc:2781] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:26.057804 17301 raft_consensus.cc:686] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [term 1 LEADER]: Becoming Leader. State: Replica: b62f619ec94747cea7280ee86234f69a, State: Running, Role: LEADER
I20260808 04:50:26.058168 17301 consensus_queue.cc:237] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } }
I20260808 04:50:26.061549 16469 catalog_manager.cc:5495] T 2031671f516345e7a6545df890b845a3 P b62f619ec94747cea7280ee86234f69a reported cstate change: term changed from 0 to 1, leader changed from <none> to b62f619ec94747cea7280ee86234f69a (127.15.30.66). New cstate: current_term: 1 leader_uuid: "b62f619ec94747cea7280ee86234f69a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b62f619ec94747cea7280ee86234f69a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 32835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79cc3bdbde9945409456138086a4759a" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e5e111a3a84d61af084fd50cbfddad" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39875 } health_report { overall_health: UNKNOWN } } }
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[16392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:26.140931 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:26.130830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.15.30.65:39875)
0808 04:50:26.131004 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:26.131009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:26.131205 (+ 196us) server_negotiation.cc:407] Connection header received
0808 04:50:26.131370 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:26.131373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:26.131425 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:26.131511 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:26.132310 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.132880 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.133881 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.134107 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.134217 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:26.137034 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:26.137052 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:26.137075 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:26.137107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:26.139171 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:26.139578 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:26.139584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:26.139585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:26.139667 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:26.139898 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:26.139900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:26.139901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:26.140137 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:26.140315 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:26.140520 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 04:50:26.140654 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[16392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[16392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:26.171006 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:26.161830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.15.30.65:39875)
0808 04:50:26.161993 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:26.161998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:26.162174 (+ 176us) server_negotiation.cc:407] Connection header received
0808 04:50:26.162359 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:26.162362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:26.162404 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:26.162488 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:26.163177 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.163701 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.164640 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.164923 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.165039 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:26.167486 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:26.167489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:26.167491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:26.167516 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:26.169084 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:26.169509 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:26.169512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:26.169513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:26.169592 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:26.169830 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:26.169833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:26.169834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:26.170092 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:26.170261 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:26.170577 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 04:50:26.170790 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[16392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[16392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:26.198762 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:26.190743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.15.30.65:39875)
0808 04:50:26.190925 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:26.190928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:26.190997 (+ 69us) server_negotiation.cc:407] Connection header received
0808 04:50:26.191102 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:26.191104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:26.191141 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:26.191212 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:26.191846 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.192331 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.193172 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:26.193387 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:26.193466 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:26.195881 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:26.195884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:26.195886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:26.195907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:26.197155 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:26.197580 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:26.197584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:26.197586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:26.197666 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:26.197943 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:26.197947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:26.197949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:26.198109 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:26.198254 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:26.198485 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 04:50:26.198599 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260808 04:50:26.200193 16654 tablet_service.cc:2908] Rejecting scan request for tablet 2031671f516345e7a6545df890b845a3: Uninitialized: clean time has not yet been initialized
I20260808 04:50:26.202128 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16584
I20260808 04:50:26.215054 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16825
I20260808 04:50:26.223374 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17050
I20260808 04:50:26.233305 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16421
2026-08-08T04:50:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2905 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[17333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[17333](info): set up 2 sockets
Aug 08 04:50:26 dist-test-slave-zpvx krb5kdc[17333](info): commencing operation
krb5kdc: starting...
W20260808 04:50:27.496059 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:27 dist-test-slave-zpvx krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:27.623240 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33333
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:27.736352 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:27.736682 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:27.736722 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:27.740746 17353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:27.740796 17353 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:50:27.740813 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:27.740831 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:27.740847 17353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:27.740862 17353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:27.746055 17353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33333
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:27.747007 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:27.748121 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:27.757306 17362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:27.757294 17363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:27.757488 17353 server_base.cc:1034] running on GCE node
W20260808 04:50:27.757294 17365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:27.758314 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:27.759464 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:27.760636 17353 hybrid_clock.cc:648] HybridClock initialized: now 1786164627760627 us; error 50 us; skew 500 ppm
Aug 08 04:50:27 dist-test-slave-zpvx krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164627, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:27.763379 17353 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:27.764215 17353 webserver.cc:460] Webserver started at http://127.15.30.126:38059/ using document root <none> and password file <none>
I20260808 04:50:27.764808 17353 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:27.764871 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:27.765036 17353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:27.767062 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "2177eab3b4cc4f39b8c411b36e4468f0"
format_stamp: "Formatted at 2026-08-08 04:50:27 on dist-test-slave-zpvx"
server_key: "9bc404bae60acda7f7ae6c4235f85b84"
server_key_iv: "c24f3b52e59453135bac0be9cfea8ef3"
server_key_version: "encryptionkey@0"
I20260808 04:50:27.767831 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "2177eab3b4cc4f39b8c411b36e4468f0"
format_stamp: "Formatted at 2026-08-08 04:50:27 on dist-test-slave-zpvx"
server_key: "9bc404bae60acda7f7ae6c4235f85b84"
server_key_iv: "c24f3b52e59453135bac0be9cfea8ef3"
server_key_version: "encryptionkey@0"
I20260808 04:50:27.771678 17353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:50:27.774171 17376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:27.775418 17353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:27.775557 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "2177eab3b4cc4f39b8c411b36e4468f0"
format_stamp: "Formatted at 2026-08-08 04:50:27 on dist-test-slave-zpvx"
server_key: "9bc404bae60acda7f7ae6c4235f85b84"
server_key_iv: "c24f3b52e59453135bac0be9cfea8ef3"
server_key_version: "encryptionkey@0"
I20260808 04:50:27.775664 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:27.789757 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:27.792519 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:27.792717 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:27.800616 17353 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33333
I20260808 04:50:27.800702 17474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33333 every 8 connection(s)
I20260808 04:50:27.801544 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:27.804293 17476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:27.809319 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17353
I20260808 04:50:27.809412 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:27.809648 15481 external_mini_cluster.cc:1428] Setting key b1ee2e90cc20e78ddd8446681fd271ae
I20260808 04:50:27.809764 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0: Bootstrap starting.
I20260808 04:50:27.812155 17476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:27.813127 17476 log.cc:824] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:27.815172 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0: No bootstrap required, opened a new log
Aug 08 04:50:27 dist-test-slave-zpvx krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:27.818997 17476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33333 } }
I20260808 04:50:27.819242 17476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:27.819322 17476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2177eab3b4cc4f39b8c411b36e4468f0, State: Initialized, Role: FOLLOWER
I20260808 04:50:27.819835 17476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33333 } }
I20260808 04:50:27.819998 17476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:27.820071 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:27.820171 17476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:27.821048 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33333 } }
I20260808 04:50:27.821381 17476 leader_election.cc:302] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2177eab3b4cc4f39b8c411b36e4468f0; no voters:
I20260808 04:50:27.821724 17476 leader_election.cc:288] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:27.821810 17483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:27.821977 17483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [term 1 LEADER]: Becoming Leader. State: Replica: 2177eab3b4cc4f39b8c411b36e4468f0, State: Running, Role: LEADER
I20260808 04:50:27.822292 17483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33333 } }
I20260808 04:50:27.822949 17476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:27.823407 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2177eab3b4cc4f39b8c411b36e4468f0. Latest consensus state: current_term: 1 leader_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33333 } } }
I20260808 04:50:27.823551 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:27.823837 17486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2177eab3b4cc4f39b8c411b36e4468f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33333 } } }
I20260808 04:50:27.823912 17489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:27.823945 17486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:27.824867 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:27.810892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.15.30.126:33333)
0808 04:50:27.811282 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:27.811298 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:27.811333 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:50:27.811945 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:27.811960 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:27.812289 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:27.812632 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:27.813571 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:27.814653 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:27.815796 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:27.816234 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:27.816811 (+ 577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:27.819159 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:27.819164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:27.819177 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:27.819202 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:27.821004 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:27.821494 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:27.821518 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:27.821527 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:27.821649 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:27.821949 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:27.821953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:27.821955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:27.822583 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:27.823483 (+ 900us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:27.823660 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 04:50:27.823997 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":138,"threads_started":1}
I20260808 04:50:27.827133 17489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:27.836792 17489 catalog_manager.cc:1347] Generated new cluster ID: a16a5492b0514e0391737570838e18d2
I20260808 04:50:27.836861 17489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:27.847385 17489 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:27.849481 17489 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:27.862337 17489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0: Generated new TSK 0
I20260808 04:50:27.863162 17489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:27.901859 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33333
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:28.013458 17515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:28.013721 17515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:28.013793 17515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:28.017730 17515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:28.017807 17515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:28.017876 17515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:50:28.017953 17515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:28.023027 17515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:28.023999 17515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:28.025153 17515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:28.034464 17523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:28.034472 17522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:28.034606 17515 server_base.cc:1034] running on GCE node
W20260808 04:50:28.034497 17525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:28.035104 17515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:28.035691 17515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:28.036856 17515 hybrid_clock.cc:648] HybridClock initialized: now 1786164628036837 us; error 31 us; skew 500 ppm
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:28.039705 17515 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:28.040620 17515 webserver.cc:460] Webserver started at http://127.15.30.65:44053/ using document root <none> and password file <none>
I20260808 04:50:28.041186 17515 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:28.041283 17515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:28.041523 17515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:28.043503 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "cd0cccee34ac47579460879f811ac657"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "d0c400e238f8ae5b8ad99c707cc312a1"
server_key_iv: "239f7f6ee168d394bb3b55d8c3d8a8a9"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.044112 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "cd0cccee34ac47579460879f811ac657"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "d0c400e238f8ae5b8ad99c707cc312a1"
server_key_iv: "239f7f6ee168d394bb3b55d8c3d8a8a9"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.047847 17515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:28.050658 17537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:28.051864 17515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:50:28.051994 17515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "cd0cccee34ac47579460879f811ac657"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "d0c400e238f8ae5b8ad99c707cc312a1"
server_key_iv: "239f7f6ee168d394bb3b55d8c3d8a8a9"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.052135 17515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:28.066354 17515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:28.069517 17515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:28.069718 17515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:28.071048 17515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:28.071132 17515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:28.071208 17515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:28.071262 17515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:28.082315 17515 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:37365
I20260808 04:50:28.082620 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:37365 every 8 connection(s)
I20260808 04:50:28.083360 17515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:28.085757 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.071978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:45015 (local address 127.15.30.126:33333)
0808 04:50:28.072140 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.072144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.073223 (+ 1079us) server_negotiation.cc:407] Connection header received
0808 04:50:28.074394 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.074401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.074476 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.074573 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.076043 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.076986 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.078346 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.078774 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.078906 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.081950 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.081954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.081957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.081987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.083536 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.084164 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.084167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.084169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.084256 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.084652 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.084657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.084659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.084844 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.084956 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.085438 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.085572 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:50:28.086439 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.072493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33333 (local address 127.15.30.65:45015)
0808 04:50:28.073067 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.073107 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.074098 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.074814 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.074825 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.075206 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:28.075863 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.075884 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.077153 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.077157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.078193 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.078206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.078947 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.078950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.079155 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.079701 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:28.079757 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:28.081758 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:28.083655 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.083665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.083681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.084034 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.084357 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.084361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.084362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.084543 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.084955 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:28.084963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:28.085197 (+ 234us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.085375 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.085677 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":134,"threads_started":1}
I20260808 04:50:28.087623 17736 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33333
I20260808 04:50:28.087783 17736 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:28.088274 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17515
I20260808 04:50:28.088351 17736 heartbeater.cc:511] Master 127.15.30.126:33333 requested a full tablet report, sending...
I20260808 04:50:28.088424 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:28.088729 15481 external_mini_cluster.cc:1428] Setting key faee2ac812d28471a0f3b65a56e9388b
I20260808 04:50:28.090022 17397 ts_manager.cc:194] Registered new tserver with Master: cd0cccee34ac47579460879f811ac657 (127.15.30.65:37365)
I20260808 04:50:28.091141 17397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:45015
I20260808 04:50:28.100082 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.089955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33333 (local address 127.15.30.65:55707)
0808 04:50:28.092600 (+ 2645us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.092618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.092723 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.092984 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.092989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.093222 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:28.093397 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.093407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.094497 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.094500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.095037 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.095044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.095690 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.095693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.095761 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.096299 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:28.096310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:28.096700 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:28.098675 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.098678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.098679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.098865 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.099165 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.099167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.099168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.099255 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.099694 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:28.099696 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:28.099767 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.099844 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.099940 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2538,"thread_start_us":96,"threads_started":1}
I20260808 04:50:28.100234 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.089884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:55707 (local address 127.15.30.126:33333)
0808 04:50:28.089997 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.090001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.092703 (+ 2702us) server_negotiation.cc:407] Connection header received
0808 04:50:28.092818 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.092821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.092867 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.092945 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.093539 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.094344 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.095170 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.095561 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.095693 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.096837 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.096844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.096850 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.096880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.098547 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.098969 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.098976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.098978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.099072 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.099346 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.099350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.099352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.099585 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.099706 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.099935 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.100051 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:28.119400 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33333
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:28.234352 17748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:28.234624 17748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:28.234699 17748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:28.238839 17748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:28.238941 17748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:28.239012 17748 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:50:28.239141 17748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:28.244424 17748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:28.245549 17748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:28.246569 17748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:28.256551 17757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:28.256564 17760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:28.256763 17748 server_base.cc:1034] running on GCE node
W20260808 04:50:28.256562 17758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:28.257225 17748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:28.257846 17748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:28.259014 17748 hybrid_clock.cc:648] HybridClock initialized: now 1786164628258982 us; error 42 us; skew 500 ppm
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:28.261734 17748 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:28.262558 17748 webserver.cc:460] Webserver started at http://127.15.30.66:40481/ using document root <none> and password file <none>
I20260808 04:50:28.263104 17748 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:28.263165 17748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:28.263397 17748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:28.265478 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "9b088df97c4a429c949ce6525d2c3346"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "084cd241acfac50de20f049d589d6f0f"
server_key_iv: "ac1cd716df4c5277fea582510ebef04e"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.266098 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "9b088df97c4a429c949ce6525d2c3346"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "084cd241acfac50de20f049d589d6f0f"
server_key_iv: "ac1cd716df4c5277fea582510ebef04e"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.269683 17748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:50:28.271991 17769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:28.273159 17748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:28.273293 17748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "9b088df97c4a429c949ce6525d2c3346"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "084cd241acfac50de20f049d589d6f0f"
server_key_iv: "ac1cd716df4c5277fea582510ebef04e"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.273398 17748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:28.296026 17748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:28.298764 17748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:28.298965 17748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:28.300359 17748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:28.300493 17748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:28.300561 17748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:28.300611 17748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:28.311620 17748 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:45577
I20260808 04:50:28.311746 17947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:45577 every 8 connection(s)
I20260808 04:50:28.312705 17748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:28.315088 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.301286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:44997 (local address 127.15.30.126:33333)
0808 04:50:28.301417 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.301420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.302457 (+ 1037us) server_negotiation.cc:407] Connection header received
0808 04:50:28.303413 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.303419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.303492 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.303605 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.305404 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.306182 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.308022 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.308334 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.308479 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.311454 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.311458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.311461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.311495 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.313078 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.313709 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.313713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.313715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.313814 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.314125 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.314128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.314130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.314331 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.314413 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.314868 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.314968 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:50:28.315929 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.301830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33333 (local address 127.15.30.66:44997)
0808 04:50:28.302314 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.302355 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.303151 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.303947 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.303960 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.304525 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:28.305214 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.305236 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.306321 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.306326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.307882 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.307896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.308623 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.308626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.308763 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.309325 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:28.309342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:28.311239 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:28.313183 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.313189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.313202 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.313529 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.313899 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.313901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.313903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.314030 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.314428 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:28.314433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:28.314644 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.314811 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.315043 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260808 04:50:28.316282 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17748
I20260808 04:50:28.316435 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:28.316777 15481 external_mini_cluster.cc:1428] Setting key 2266f86b86d0ef27c8252eb772b74525
I20260808 04:50:28.319092 17950 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33333
I20260808 04:50:28.319283 17950 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:28.319758 17950 heartbeater.cc:511] Master 127.15.30.126:33333 requested a full tablet report, sending...
I20260808 04:50:28.320837 17398 ts_manager.cc:194] Registered new tserver with Master: 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66:45577)
I20260808 04:50:28.321422 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:44997
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:50:28.328996 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.319315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33333 (local address 127.15.30.66:46415)
0808 04:50:28.320524 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.320541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.320689 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.320975 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.320980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.321227 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:28.321371 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.321379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.322337 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.322340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.323174 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.323185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.324201 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.324204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.324277 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.324914 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:28.324926 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:28.325247 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:28.327307 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.327309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.327311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.327558 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.327911 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.327913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.327914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.327993 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.328523 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:28.328525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:28.328602 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.328720 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.328851 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"thread_start_us":119,"threads_started":1}
I20260808 04:50:28.329131 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.319426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:46415 (local address 127.15.30.126:33333)
0808 04:50:28.319541 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.319545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.320615 (+ 1070us) server_negotiation.cc:407] Connection header received
0808 04:50:28.320794 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.320798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.320844 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.320934 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.321504 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.322189 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.323629 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.324096 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.324205 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.325394 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.325400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.325404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.325436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.327181 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.327700 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.327704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.327706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.327807 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.328094 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.328099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.328104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.328425 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.328524 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.328808 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.328936 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:28.348268 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33333
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:28.462447 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:28.462742 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:28.462855 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:28.467530 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:28.467612 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:28.467679 17963 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:50:28.467741 17963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:28.472939 17963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:33333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:28.474047 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:28.475057 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:28.485147 17979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:28.485141 17976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:28.485293 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:28.485427 17963 server_base.cc:1034] running on GCE node
I20260808 04:50:28.485795 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:28.486423 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:28.487605 17963 hybrid_clock.cc:648] HybridClock initialized: now 1786164628487563 us; error 54 us; skew 500 ppm
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:28.490576 17963 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:28.491380 17963 webserver.cc:460] Webserver started at http://127.15.30.67:41095/ using document root <none> and password file <none>
I20260808 04:50:28.491910 17963 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:28.491969 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:28.492132 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:28.494158 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "b96c108fc91f4bebb55071a37a9cd280"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "d256af9d90ce315299b39bb8929163f1"
server_key_iv: "ddcf9b537bfd579252e6aff79404c57a"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.494686 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "b96c108fc91f4bebb55071a37a9cd280"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "d256af9d90ce315299b39bb8929163f1"
server_key_iv: "ddcf9b537bfd579252e6aff79404c57a"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.498668 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:50:28.501137 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:28.502444 17963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:28.502600 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "b96c108fc91f4bebb55071a37a9cd280"
format_stamp: "Formatted at 2026-08-08 04:50:28 on dist-test-slave-zpvx"
server_key: "d256af9d90ce315299b39bb8929163f1"
server_key_iv: "ddcf9b537bfd579252e6aff79404c57a"
server_key_version: "encryptionkey@0"
I20260808 04:50:28.502717 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:28.526914 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:28.530241 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:28.530465 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:28.532027 17963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:28.532109 17963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:28.532186 17963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:28.532239 17963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:28.544966 17963 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:41285
I20260808 04:50:28.545133 18167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:41285 every 8 connection(s)
I20260808 04:50:28.546145 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:28.548035 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.533389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33333 (local address 127.15.30.67:59405)
0808 04:50:28.533894 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.533957 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.535216 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.535939 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.535945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.536305 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:28.536845 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.536860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.537681 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.537686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.538718 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.538733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.539225 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.539229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.539397 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.540021 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:28.540044 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:28.543115 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:28.545168 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.545174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.545189 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.545486 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.545835 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.545837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.545839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.545968 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.546455 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:28.546460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:28.546717 (+ 257us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.546925 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.547288 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260808 04:50:28.548715 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.532981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:59405 (local address 127.15.30.126:33333)
0808 04:50:28.533150 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.533156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.534091 (+ 935us) server_negotiation.cc:407] Connection header received
0808 04:50:28.535552 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.535559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.535633 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.535761 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.537016 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.537550 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.538841 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.539068 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.539181 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.543289 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.543293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.543295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.543330 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.544989 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.545604 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.545608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.545611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.545726 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.546075 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.546082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.546087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.546345 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.546466 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.548500 (+ 2034us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.548602 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:50:28.549777 18170 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33333
I20260808 04:50:28.549942 18170 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:28.550621 18170 heartbeater.cc:511] Master 127.15.30.126:33333 requested a full tablet report, sending...
I20260808 04:50:28.551744 17397 ts_manager.cc:194] Registered new tserver with Master: b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67:41285)
I20260808 04:50:28.552546 17397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:59405
I20260808 04:50:28.555372 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17963
I20260808 04:50:28.555524 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:28.555922 15481 external_mini_cluster.cc:1428] Setting key f87c85b7bae41b78b399b192b8bb49db
I20260808 04:50:28.558550 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:50:28.559712 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.551976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33333 (local address 127.15.30.67:58473)
0808 04:50:28.552260 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.552272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.552401 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.552631 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.552633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.552833 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:28.552959 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.552965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.553984 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.553986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.554536 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.554544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.555153 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.555160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.555268 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.555821 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:28.555838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:28.556245 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:28.558120 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.558123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.558125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.558348 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.558677 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.558682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.558684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.558776 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.559240 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:28.559242 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:28.559317 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.559401 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.559508 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":119,"threads_started":1}
I20260808 04:50:28.559855 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.551904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:58473 (local address 127.15.30.126:33333)
0808 04:50:28.551995 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.551998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.552346 (+ 348us) server_negotiation.cc:407] Connection header received
0808 04:50:28.552487 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.552490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.552533 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.552614 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.553062 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.553858 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.554656 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.554961 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.555052 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.556390 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.556394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.556396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.556420 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.557997 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.558449 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.558452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.558470 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.558569 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.558868 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.558871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.558872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.559091 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.559240 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.559492 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.559629 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:28.586731 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.578109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.15.30.126:33333)
0808 04:50:28.578260 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.578264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.578278 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:50:28.578316 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.578318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.578356 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.578488 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.579193 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.579754 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.580929 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.581166 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.581258 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.583638 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.583642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.583644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.583666 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.585221 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.585624 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.585626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.585628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.585708 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.585994 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.585996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.585997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.586168 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.586301 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.586512 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.586606 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:50:28.589903 17397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:28.593009 17397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:28.607059 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.601756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.15.30.65:37365)
0808 04:50:28.602358 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.602366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.602394 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:50:28.602483 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.602489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.602721 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.602897 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.603217 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.604129 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.605554 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.606378 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.606516 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.606699 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.606832 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.606927 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":86,"threads_started":1}
I20260808 04:50:28.607059 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.601620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:37365 (local address 127.0.0.1:49712)
0808 04:50:28.601969 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.602026 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.602170 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.602904 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.602909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.602929 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.603050 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.603056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.604277 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.604280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.605418 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.605429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.606538 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.606541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.606647 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.606686 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.606801 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.606921 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260808 04:50:28.608820 18210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.602651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41285 (local address 127.0.0.1:42952)
0808 04:50:28.603565 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.603583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.603671 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.604412 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.604415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.604433 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.604506 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.604511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.606225 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.606229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.607315 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.607327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.608411 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.608416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.608501 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.608529 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.608620 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.608694 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260808 04:50:28.608870 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.603287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.15.30.67:41285)
0808 04:50:28.603949 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.603963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.603985 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:50:28.604062 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.604066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.604246 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.604451 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.604592 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.606086 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.607515 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.608237 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.608365 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.608543 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.608648 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.608733 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":103,"threads_started":1}
I20260808 04:50:28.610602 18208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.602339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:45577 (local address 127.0.0.1:50706)
0808 04:50:28.602668 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.602685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.602776 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.604000 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.604003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.604021 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.604113 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.604119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.605591 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.605594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.607791 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.607801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.610212 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.610216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.610304 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.610332 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.610410 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.610475 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":1002,"thread_start_us":115,"threads_started":1}
I20260808 04:50:28.610543 17622 tablet_service.cc:1467] Processing CreateTablet for tablet a5ec560879bf47ababe147110158f34a (DEFAULT_TABLE table=test-table [id=fe24a1e803de479b89422ea2687ff360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:28.611071 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.602580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.15.30.66:45577)
0808 04:50:28.603510 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.603525 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.603549 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:50:28.603636 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.603641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.603847 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.604011 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.604222 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.605494 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.607916 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.608852 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.609819 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.609985 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.610839 (+ 854us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.610936 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":102,"threads_started":1}
I20260808 04:50:28.611794 17622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ec560879bf47ababe147110158f34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.611788 18065 tablet_service.cc:1467] Processing CreateTablet for tablet a5ec560879bf47ababe147110158f34a (DEFAULT_TABLE table=test-table [id=fe24a1e803de479b89422ea2687ff360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:28.612876 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ec560879bf47ababe147110158f34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.613832 17845 tablet_service.cc:1467] Processing CreateTablet for tablet a5ec560879bf47ababe147110158f34a (DEFAULT_TABLE table=test-table [id=fe24a1e803de479b89422ea2687ff360]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:28.614787 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ec560879bf47ababe147110158f34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.619671 18213 tablet_bootstrap.cc:492] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Bootstrap starting.
I20260808 04:50:28.620637 18214 tablet_bootstrap.cc:492] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Bootstrap starting.
I20260808 04:50:28.622068 18216 tablet_bootstrap.cc:492] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Bootstrap starting.
I20260808 04:50:28.622859 18213 tablet_bootstrap.cc:654] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.623268 18214 tablet_bootstrap.cc:654] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.623876 18213 log.cc:824] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:28.624177 18214 log.cc:824] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:28.624308 18216 tablet_bootstrap.cc:654] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.625252 18216 log.cc:824] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:28.626416 18214 tablet_bootstrap.cc:492] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: No bootstrap required, opened a new log
I20260808 04:50:28.626471 18213 tablet_bootstrap.cc:492] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: No bootstrap required, opened a new log
I20260808 04:50:28.626631 18214 ts_tablet_manager.cc:1397] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:50:28.626665 18213 ts_tablet_manager.cc:1397] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:50:28.627139 18216 tablet_bootstrap.cc:492] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: No bootstrap required, opened a new log
I20260808 04:50:28.627305 18216 ts_tablet_manager.cc:1397] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:50:28.629626 18214 raft_consensus.cc:348] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.629858 18214 raft_consensus.cc:374] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.629931 18214 raft_consensus.cc:729] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96c108fc91f4bebb55071a37a9cd280, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.630384 18214 consensus_queue.cc:260] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.631029 18170 heartbeater.cc:503] Master 127.15.30.126:33333 was elected leader, sending a full tablet report...
I20260808 04:50:28.631234 18213 raft_consensus.cc:348] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.631234 18216 raft_consensus.cc:348] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.631469 18216 raft_consensus.cc:374] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.631469 18213 raft_consensus.cc:374] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.631518 18213 raft_consensus.cc:729] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0cccee34ac47579460879f811ac657, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.631518 18216 raft_consensus.cc:729] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b088df97c4a429c949ce6525d2c3346, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.631920 18216 consensus_queue.cc:260] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.631920 18213 consensus_queue.cc:260] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.632503 18214 ts_tablet_manager.cc:1428] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:50:28.632865 18216 ts_tablet_manager.cc:1428] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:50:28.632865 18213 ts_tablet_manager.cc:1428] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:50:28.633222 17950 heartbeater.cc:503] Master 127.15.30.126:33333 was elected leader, sending a full tablet report...
I20260808 04:50:28.634209 17736 heartbeater.cc:503] Master 127.15.30.126:33333 was elected leader, sending a full tablet report...
I20260808 04:50:28.641563 18223 raft_consensus.cc:482] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:28.641711 18223 raft_consensus.cc:504] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.642659 18223 leader_election.cc:288] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd0cccee34ac47579460879f811ac657 (127.15.30.65:37365), 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66:45577)
I20260808 04:50:28.647027 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.642835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:37365 (local address 127.15.30.67:59383)
0808 04:50:28.642979 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.642996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.643124 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.643396 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.643398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.643415 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.643563 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.643572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.644676 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.644678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.645917 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.645927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.646705 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.646707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.646791 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.646811 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.646873 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.646929 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 04:50:28.647126 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.642904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:59383 (local address 127.15.30.65:37365)
0808 04:50:28.643034 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.643041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.643056 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:28.643242 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.643245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.643296 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.643375 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.643702 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.644536 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.646075 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.646569 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.646689 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.646737 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.646910 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.646982 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:50:28.647653 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.643510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:33697 (local address 127.15.30.66:45577)
0808 04:50:28.643673 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.643679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.643692 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:50:28.643733 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.643736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.643778 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.643858 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.644088 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.644820 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.646449 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.647031 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.647145 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.647202 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.647490 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.647565 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:50:28.647691 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.643178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:45577 (local address 127.15.30.67:33697)
0808 04:50:28.643449 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.643465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.643561 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.643862 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.643864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.643875 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.643978 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.643983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.644916 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.644919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.646326 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.646334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.647268 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.647272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.647367 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.647384 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.647472 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.647540 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
I20260808 04:50:28.647738 17659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ec560879bf47ababe147110158f34a" candidate_uuid: "b96c108fc91f4bebb55071a37a9cd280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0cccee34ac47579460879f811ac657" is_pre_election: true
I20260808 04:50:28.648041 17659 raft_consensus.cc:2454] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b96c108fc91f4bebb55071a37a9cd280 in term 0.
I20260808 04:50:28.648339 17874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ec560879bf47ababe147110158f34a" candidate_uuid: "b96c108fc91f4bebb55071a37a9cd280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b088df97c4a429c949ce6525d2c3346" is_pre_election: true
I20260808 04:50:28.648569 17994 leader_election.cc:302] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b96c108fc91f4bebb55071a37a9cd280, cd0cccee34ac47579460879f811ac657; no voters:
I20260808 04:50:28.648661 17874 raft_consensus.cc:2454] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b96c108fc91f4bebb55071a37a9cd280 in term 0.
I20260808 04:50:28.648842 18223 raft_consensus.cc:2781] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:28.648932 18223 raft_consensus.cc:482] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:28.648995 18223 raft_consensus.cc:3037] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.650187 18223 raft_consensus.cc:504] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.650549 18223 leader_election.cc:288] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [CANDIDATE]: Term 1 election: Requested vote from peers cd0cccee34ac47579460879f811ac657 (127.15.30.65:37365), 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66:45577)
I20260808 04:50:28.650907 17659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ec560879bf47ababe147110158f34a" candidate_uuid: "b96c108fc91f4bebb55071a37a9cd280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0cccee34ac47579460879f811ac657"
I20260808 04:50:28.650931 17874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ec560879bf47ababe147110158f34a" candidate_uuid: "b96c108fc91f4bebb55071a37a9cd280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b088df97c4a429c949ce6525d2c3346"
I20260808 04:50:28.651041 17659 raft_consensus.cc:3037] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.651077 17874 raft_consensus.cc:3037] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.652103 17659 raft_consensus.cc:2454] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b96c108fc91f4bebb55071a37a9cd280 in term 1.
I20260808 04:50:28.652395 17874 raft_consensus.cc:2454] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b96c108fc91f4bebb55071a37a9cd280 in term 1.
I20260808 04:50:28.652473 17994 leader_election.cc:302] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b96c108fc91f4bebb55071a37a9cd280, cd0cccee34ac47579460879f811ac657; no voters:
I20260808 04:50:28.652655 18223 raft_consensus.cc:2781] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:28.652889 18223 raft_consensus.cc:686] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 1 LEADER]: Becoming Leader. State: Replica: b96c108fc91f4bebb55071a37a9cd280, State: Running, Role: LEADER
I20260808 04:50:28.653225 18223 consensus_queue.cc:237] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.656464 17397 catalog_manager.cc:5495] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 reported cstate change: term changed from 0 to 1, leader changed from <none> to b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67). New cstate: current_term: 1 leader_uuid: "b96c108fc91f4bebb55071a37a9cd280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } health_report { overall_health: HEALTHY } } }
I20260808 04:50:28.676225 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.672461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.15.30.67:41285)
0808 04:50:28.672585 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.672588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.672602 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:50:28.672758 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.672761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.672815 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.672901 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:28.673271 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.673783 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.674667 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.674882 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.674962 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.675298 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:28.675706 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:28.675843 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.676055 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.676112 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:50:28.682734 17659 raft_consensus.cc:1261] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Refusing update from remote peer b96c108fc91f4bebb55071a37a9cd280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:28.683458 17874 raft_consensus.cc:1261] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Refusing update from remote peer b96c108fc91f4bebb55071a37a9cd280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:28.683466 18230 consensus_queue.cc:1035] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:28.684137 18230 consensus_queue.cc:1035] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:28.693176 18234 mvcc.cc:205] Tried to move back new op lower bound from 7316130319075942400 to 7316130318966353920. Current Snapshot: MvccSnapshot[applied={T|T < 7316130319075942400}]
I20260808 04:50:28.693840 18233 mvcc.cc:205] Tried to move back new op lower bound from 7316130319075942400 to 7316130318966353920. Current Snapshot: MvccSnapshot[applied={T|T < 7316130319075942400}]
I20260808 04:50:28.695161 18235 mvcc.cc:205] Tried to move back new op lower bound from 7316130319075942400 to 7316130318966353920. Current Snapshot: MvccSnapshot[applied={T|T < 7316130319075942400}]
I20260808 04:50:28.706202 17397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:28.706458 17397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:28.709456 17397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0: Sending DeleteTablet for 3 replicas of tablet a5ec560879bf47ababe147110158f34a
I20260808 04:50:28.710515 17622 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ec560879bf47ababe147110158f34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:28 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:49712
I20260808 04:50:28.710563 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ec560879bf47ababe147110158f34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:28 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:50706
I20260808 04:50:28.710640 18065 tablet_service.cc:1514] Processing DeleteTablet for tablet a5ec560879bf47ababe147110158f34a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:28 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42952
I20260808 04:50:28.711259 18247 tablet_replica.cc:330] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: stopping tablet replica
I20260808 04:50:28.711249 18248 tablet_replica.cc:330] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: stopping tablet replica
I20260808 04:50:28.711609 18247 raft_consensus.cc:2229] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:28.711637 18248 raft_consensus.cc:2229] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:28.712037 18248 raft_consensus.cc:2258] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:28.712044 18247 raft_consensus.cc:2258] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:28.712519 18246 tablet_replica.cc:330] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: stopping tablet replica
I20260808 04:50:28.712419 17397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:50:28.712831 18246 raft_consensus.cc:2229] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 04:50:28.712904 17397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:28.713140 18246 raft_consensus.cc:2258] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:28.714283 18248 ts_tablet_manager.cc:1905] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:28.715587 18247 ts_tablet_manager.cc:1905] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:28.717608 18248 ts_tablet_manager.cc:1918] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:28.717722 18248 log.cc:1196] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/a5ec560879bf47ababe147110158f34a
I20260808 04:50:28.718051 18248 ts_tablet_manager.cc:1939] T a5ec560879bf47ababe147110158f34a P 9b088df97c4a429c949ce6525d2c3346: Deleting consensus metadata
I20260808 04:50:28.719340 17622 tablet_service.cc:1467] Processing CreateTablet for tablet 48acf885df0a42f69045b80e8b205296 (DEFAULT_TABLE table=test-table [id=afe6ec7db0b74b3bbfd284923e475809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:28.719381 17380 catalog_manager.cc:4841] TS 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66:45577): tablet a5ec560879bf47ababe147110158f34a (table test-table [id=fe24a1e803de479b89422ea2687ff360]) successfully deleted
I20260808 04:50:28.719657 17622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48acf885df0a42f69045b80e8b205296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.719830 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 48acf885df0a42f69045b80e8b205296 (DEFAULT_TABLE table=test-table [id=afe6ec7db0b74b3bbfd284923e475809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:28.719889 18247 ts_tablet_manager.cc:1918] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:28.720081 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48acf885df0a42f69045b80e8b205296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.720170 18247 log.cc:1196] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/a5ec560879bf47ababe147110158f34a
I20260808 04:50:28.720661 18247 ts_tablet_manager.cc:1939] T a5ec560879bf47ababe147110158f34a P b96c108fc91f4bebb55071a37a9cd280: Deleting consensus metadata
I20260808 04:50:28.722644 17379 catalog_manager.cc:4841] TS b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67:41285): tablet a5ec560879bf47ababe147110158f34a (table test-table [id=fe24a1e803de479b89422ea2687ff360]) successfully deleted
I20260808 04:50:28.722731 18216 tablet_bootstrap.cc:492] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Bootstrap starting.
I20260808 04:50:28.723435 18065 tablet_service.cc:1467] Processing CreateTablet for tablet 48acf885df0a42f69045b80e8b205296 (DEFAULT_TABLE table=test-table [id=afe6ec7db0b74b3bbfd284923e475809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:28.723803 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48acf885df0a42f69045b80e8b205296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.724244 18216 tablet_bootstrap.cc:654] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.724340 18246 ts_tablet_manager.cc:1905] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:28.726779 18213 tablet_bootstrap.cc:492] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Bootstrap starting.
I20260808 04:50:28.726980 18216 tablet_bootstrap.cc:492] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: No bootstrap required, opened a new log
I20260808 04:50:28.727061 18216 ts_tablet_manager.cc:1397] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260808 04:50:28.727797 18246 ts_tablet_manager.cc:1918] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:28.727883 18246 log.cc:1196] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/a5ec560879bf47ababe147110158f34a
I20260808 04:50:28.728176 18246 ts_tablet_manager.cc:1939] T a5ec560879bf47ababe147110158f34a P cd0cccee34ac47579460879f811ac657: Deleting consensus metadata
I20260808 04:50:28.728111 18216 raft_consensus.cc:348] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.728258 18216 raft_consensus.cc:374] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.728298 18216 raft_consensus.cc:729] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b088df97c4a429c949ce6525d2c3346, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.728648 18213 tablet_bootstrap.cc:654] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.729478 17380 catalog_manager.cc:4841] TS cd0cccee34ac47579460879f811ac657 (127.15.30.65:37365): tablet a5ec560879bf47ababe147110158f34a (table test-table [id=fe24a1e803de479b89422ea2687ff360]) successfully deleted
I20260808 04:50:28.729508 18216 consensus_queue.cc:260] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.729856 18216 ts_tablet_manager.cc:1428] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 04:50:28.729936 18214 tablet_bootstrap.cc:492] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Bootstrap starting.
I20260808 04:50:28.730872 18213 tablet_bootstrap.cc:492] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: No bootstrap required, opened a new log
I20260808 04:50:28.730944 18213 ts_tablet_manager.cc:1397] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 04:50:28.731271 18214 tablet_bootstrap.cc:654] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.731694 18213 raft_consensus.cc:348] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.731807 18213 raft_consensus.cc:374] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.731837 18213 raft_consensus.cc:729] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0cccee34ac47579460879f811ac657, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.731980 18213 consensus_queue.cc:260] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.732290 18213 ts_tablet_manager.cc:1428] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:50:28.733697 18214 tablet_bootstrap.cc:492] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: No bootstrap required, opened a new log
I20260808 04:50:28.733790 18214 ts_tablet_manager.cc:1397] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 04:50:28.734572 18214 raft_consensus.cc:348] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.734697 18214 raft_consensus.cc:374] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.734730 18214 raft_consensus.cc:729] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96c108fc91f4bebb55071a37a9cd280, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.734850 18214 consensus_queue.cc:260] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.735109 18214 ts_tablet_manager.cc:1428] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:50:28.782579 18226 raft_consensus.cc:482] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:28.782796 18226 raft_consensus.cc:504] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.784158 18226 leader_election.cc:288] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd0cccee34ac47579460879f811ac657 (127.15.30.65:37365), b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67:41285)
I20260808 04:50:28.789296 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.784619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:46329 (local address 127.15.30.65:37365)
0808 04:50:28.784892 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.784900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.784916 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:50:28.784974 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.784977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.785028 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.785191 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.785481 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.786205 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.787836 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.788560 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.788713 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.788782 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.789069 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.789153 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260808 04:50:28.789336 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.784461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:37365 (local address 127.15.30.66:46329)
0808 04:50:28.784689 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.784706 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.784809 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.785200 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.785204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.785228 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.785355 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.785363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.786328 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.786332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.787643 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.787655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.788783 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.788788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.788910 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.788939 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.789043 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.789122 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":3}
I20260808 04:50:28.789857 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.784710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:44915 (local address 127.15.30.67:41285)
0808 04:50:28.785028 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.785033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.785208 (+ 175us) server_negotiation.cc:407] Connection header received
0808 04:50:28.785436 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.785441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.785496 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.785578 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.786018 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.786642 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.788320 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.789109 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.789227 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.789277 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.789647 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.789728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260808 04:50:28.789963 17659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48acf885df0a42f69045b80e8b205296" candidate_uuid: "9b088df97c4a429c949ce6525d2c3346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0cccee34ac47579460879f811ac657" is_pre_election: true
I20260808 04:50:28.790122 17659 raft_consensus.cc:2454] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b088df97c4a429c949ce6525d2c3346 in term 0.
I20260808 04:50:28.790403 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.784461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41285 (local address 127.15.30.66:44915)
0808 04:50:28.785065 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.785080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.785296 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.785601 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.785604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.785623 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.785879 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.785888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.786771 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.786777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.788185 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.788195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.789348 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.789355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.789473 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.789504 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.790109 (+ 605us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.790206 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":216,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
I20260808 04:50:28.790547 17773 leader_election.cc:302] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b088df97c4a429c949ce6525d2c3346, cd0cccee34ac47579460879f811ac657; no voters:
I20260808 04:50:28.790859 18226 raft_consensus.cc:2781] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:28.790946 18226 raft_consensus.cc:482] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:28.791004 18226 raft_consensus.cc:3037] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.791213 18104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48acf885df0a42f69045b80e8b205296" candidate_uuid: "9b088df97c4a429c949ce6525d2c3346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96c108fc91f4bebb55071a37a9cd280" is_pre_election: true
I20260808 04:50:28.791502 18104 raft_consensus.cc:2454] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b088df97c4a429c949ce6525d2c3346 in term 0.
I20260808 04:50:28.792896 18226 raft_consensus.cc:504] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.793287 18226 leader_election.cc:288] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [CANDIDATE]: Term 1 election: Requested vote from peers cd0cccee34ac47579460879f811ac657 (127.15.30.65:37365), b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67:41285)
I20260808 04:50:28.793653 18104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48acf885df0a42f69045b80e8b205296" candidate_uuid: "9b088df97c4a429c949ce6525d2c3346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96c108fc91f4bebb55071a37a9cd280"
I20260808 04:50:28.793684 17659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48acf885df0a42f69045b80e8b205296" candidate_uuid: "9b088df97c4a429c949ce6525d2c3346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0cccee34ac47579460879f811ac657"
I20260808 04:50:28.793802 18104 raft_consensus.cc:3037] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.793826 17659 raft_consensus.cc:3037] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.794963 17659 raft_consensus.cc:2454] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b088df97c4a429c949ce6525d2c3346 in term 1.
I20260808 04:50:28.795279 18104 raft_consensus.cc:2454] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b088df97c4a429c949ce6525d2c3346 in term 1.
I20260808 04:50:28.795701 17773 leader_election.cc:302] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b088df97c4a429c949ce6525d2c3346, cd0cccee34ac47579460879f811ac657; no voters:
I20260808 04:50:28.795990 18226 raft_consensus.cc:2781] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:28.796198 18226 raft_consensus.cc:686] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 1 LEADER]: Becoming Leader. State: Replica: 9b088df97c4a429c949ce6525d2c3346, State: Running, Role: LEADER
I20260808 04:50:28.796481 18226 consensus_queue.cc:237] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
W20260808 04:50:28.797237 18171 tablet.cc:2369] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:28.799305 17398 catalog_manager.cc:5495] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66). New cstate: current_term: 1 leader_uuid: "9b088df97c4a429c949ce6525d2c3346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } health_report { overall_health: UNKNOWN } } }
W20260808 04:50:28.813997 17951 tablet.cc:2369] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:50:28.834833 17737 tablet.cc:2369] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:50:28 dist-test-slave-zpvx krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164627, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:28.840829 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.830249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33333 (local address 127.0.0.1:46394)
0808 04:50:28.830692 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.830706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.830866 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.831165 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.831168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.831626 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:28.831785 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.831798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.833043 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.833046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.833635 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.833643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.834323 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.834326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.834436 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.835151 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:28.835165 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:28.836952 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:28.839182 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.839191 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.839196 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.839427 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.839776 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:28.839779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:28.839780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:28.839866 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:28.840313 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:28.840318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:28.840472 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.840578 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.840691 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":33,"thread_start_us":227,"threads_started":1}
I20260808 04:50:28.840965 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.830323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.15.30.126:33333)
0808 04:50:28.830472 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.830476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.830804 (+ 328us) server_negotiation.cc:407] Connection header received
0808 04:50:28.830969 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.830972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.831041 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.831134 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:28.831935 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.832901 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.833788 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.834179 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.834305 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.837143 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:28.837148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:28.837151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:28.837197 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:28.839068 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.839536 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.839542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.839543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.839656 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:28.839965 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:28.839968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:28.839970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:28.840202 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:28.840310 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.840657 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.840785 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
I20260808 04:50:28.845431 17398 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:46394:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:50:28.846191 17398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:28.851811 17845 tablet_service.cc:1467] Processing CreateTablet for tablet d3a96103e9c74710ab2c70deeb8c519d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa0d7511259e4ea080fe37408e91d570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:50:28.851917 17622 tablet_service.cc:1467] Processing CreateTablet for tablet d3a96103e9c74710ab2c70deeb8c519d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa0d7511259e4ea080fe37408e91d570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:50:28.852319 17622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a96103e9c74710ab2c70deeb8c519d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.852325 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a96103e9c74710ab2c70deeb8c519d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.852928 18065 tablet_service.cc:1467] Processing CreateTablet for tablet d3a96103e9c74710ab2c70deeb8c519d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa0d7511259e4ea080fe37408e91d570]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:50:28.853384 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a96103e9c74710ab2c70deeb8c519d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:28.855521 18213 tablet_bootstrap.cc:492] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: Bootstrap starting.
I20260808 04:50:28.856717 18213 tablet_bootstrap.cc:654] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.857336 18214 tablet_bootstrap.cc:492] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: Bootstrap starting.
I20260808 04:50:28.857642 18216 tablet_bootstrap.cc:492] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: Bootstrap starting.
I20260808 04:50:28.858461 18213 tablet_bootstrap.cc:492] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: No bootstrap required, opened a new log
I20260808 04:50:28.858541 18213 ts_tablet_manager.cc:1397] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:28.858876 18216 tablet_bootstrap.cc:654] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.859210 18214 tablet_bootstrap.cc:654] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:28.859228 18213 raft_consensus.cc:348] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.859393 18213 raft_consensus.cc:374] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.859427 18213 raft_consensus.cc:729] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0cccee34ac47579460879f811ac657, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.859557 18213 consensus_queue.cc:260] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.859867 18213 ts_tablet_manager.cc:1428] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:50:28.860152 18224 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.860467 18224 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:28.861049 18216 tablet_bootstrap.cc:492] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: No bootstrap required, opened a new log
I20260808 04:50:28.861150 18216 ts_tablet_manager.cc:1397] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 04:50:28.861666 18216 raft_consensus.cc:348] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.861794 18216 raft_consensus.cc:374] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.861832 18216 raft_consensus.cc:729] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b088df97c4a429c949ce6525d2c3346, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.861961 18216 consensus_queue.cc:260] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.862289 18216 ts_tablet_manager.cc:1428] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:50:28.862294 18226 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.862481 18226 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:28.863507 18214 tablet_bootstrap.cc:492] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: No bootstrap required, opened a new log
I20260808 04:50:28.863606 18214 ts_tablet_manager.cc:1397] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260808 04:50:28.864260 18214 raft_consensus.cc:348] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.864410 18214 raft_consensus.cc:374] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:28.864449 18214 raft_consensus.cc:729] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96c108fc91f4bebb55071a37a9cd280, State: Initialized, Role: FOLLOWER
I20260808 04:50:28.864606 18214 consensus_queue.cc:260] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.864871 18223 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.865039 18223 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:28.865658 18214 ts_tablet_manager.cc:1428] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 04:50:28.938305 18224 raft_consensus.cc:482] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:28.938486 18224 raft_consensus.cc:504] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.939623 18224 leader_election.cc:288] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66:45577), b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67:41285)
I20260808 04:50:28.944049 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.940121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:52469 (local address 127.15.30.66:45577)
0808 04:50:28.940282 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.940287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.940635 (+ 348us) server_negotiation.cc:407] Connection header received
0808 04:50:28.940805 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.940808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.940859 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.940946 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.941336 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.941829 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.943069 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.943465 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.943596 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.943659 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.943866 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.943939 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:50:28.944077 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.939960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:45577 (local address 127.15.30.65:52469)
0808 04:50:28.940519 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.940531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.940671 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.940958 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.940961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.940978 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.941203 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.941210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.941947 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.941952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.942928 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.942938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.943593 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.943596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.943727 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.943748 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.943841 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.943909 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":65,"thread_start_us":249,"threads_started":1}
I20260808 04:50:28.944181 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.940019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:45987 (local address 127.15.30.67:41285)
0808 04:50:28.940205 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.940210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.940568 (+ 358us) server_negotiation.cc:407] Connection header received
0808 04:50:28.940737 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.940740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.940790 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.940873 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:28.941253 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.941773 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.943133 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.943532 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.943675 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.943728 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.943970 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.944041 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:50:28.944643 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.939893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41285 (local address 127.15.30.65:45987)
0808 04:50:28.940459 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.940472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.940615 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.940886 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.940889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.940922 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:28.941078 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.941084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.941890 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.941895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.942996 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.943006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.943661 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.943665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.943836 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.943854 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.943949 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.944017 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":71,"thread_start_us":212,"threads_started":1}
I20260808 04:50:28.944650 17874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a96103e9c74710ab2c70deeb8c519d" candidate_uuid: "cd0cccee34ac47579460879f811ac657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b088df97c4a429c949ce6525d2c3346" is_pre_election: true
I20260808 04:50:28.944826 17874 raft_consensus.cc:2454] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd0cccee34ac47579460879f811ac657 in term 0.
I20260808 04:50:28.945125 18104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a96103e9c74710ab2c70deeb8c519d" candidate_uuid: "cd0cccee34ac47579460879f811ac657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96c108fc91f4bebb55071a37a9cd280" is_pre_election: true
I20260808 04:50:28.945250 18104 raft_consensus.cc:2454] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd0cccee34ac47579460879f811ac657 in term 0.
I20260808 04:50:28.945263 17548 leader_election.cc:302] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b088df97c4a429c949ce6525d2c3346, cd0cccee34ac47579460879f811ac657; no voters:
I20260808 04:50:28.945536 18224 raft_consensus.cc:2781] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:28.945637 18224 raft_consensus.cc:482] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:28.945705 18224 raft_consensus.cc:3037] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.947042 18224 raft_consensus.cc:504] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.947434 18224 leader_election.cc:288] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [CANDIDATE]: Term 1 election: Requested vote from peers 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66:45577), b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67:41285)
I20260808 04:50:28.947778 17874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a96103e9c74710ab2c70deeb8c519d" candidate_uuid: "cd0cccee34ac47579460879f811ac657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b088df97c4a429c949ce6525d2c3346"
I20260808 04:50:28.947853 18104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a96103e9c74710ab2c70deeb8c519d" candidate_uuid: "cd0cccee34ac47579460879f811ac657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96c108fc91f4bebb55071a37a9cd280"
I20260808 04:50:28.947918 17874 raft_consensus.cc:3037] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.947991 18104 raft_consensus.cc:3037] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:28.948951 17874 raft_consensus.cc:2454] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd0cccee34ac47579460879f811ac657 in term 1.
I20260808 04:50:28.949175 18104 raft_consensus.cc:2454] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd0cccee34ac47579460879f811ac657 in term 1.
I20260808 04:50:28.949461 17548 leader_election.cc:302] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b088df97c4a429c949ce6525d2c3346, cd0cccee34ac47579460879f811ac657; no voters:
I20260808 04:50:28.949649 18224 raft_consensus.cc:2781] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:28.949965 18224 raft_consensus.cc:686] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [term 1 LEADER]: Becoming Leader. State: Replica: cd0cccee34ac47579460879f811ac657, State: Running, Role: LEADER
I20260808 04:50:28.950259 18224 consensus_queue.cc:237] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } }
I20260808 04:50:28.952135 18278 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: TxnStatusTablet state changed. Reason: New leader cd0cccee34ac47579460879f811ac657. Latest consensus state: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.952257 18278 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:50:28.952610 18280 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:50:28.952772 18280 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:50:28.953084 17397 catalog_manager.cc:5495] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd0cccee34ac47579460879f811ac657 (127.15.30.65). New cstate: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } health_report { overall_health: UNKNOWN } } }
I20260808 04:50:28.968294 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.963703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:37365 (local address 127.0.0.1:49726)
0808 04:50:28.963869 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.963886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.963995 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.964356 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.964358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.964402 (+ 44us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:28.964576 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.964586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.965762 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.965765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.966395 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.966404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.966998 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.967001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.967100 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.967144 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:28.968013 (+ 869us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:28.968028 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.968118 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.968189 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 04:50:28.968333 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.963834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.15.30.65:37365)
0808 04:50:28.963942 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.963946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.963961 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:28.964158 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.964162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.964228 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.964332 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:28.964704 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.965577 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.966523 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.966883 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.966994 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.967305 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:28.967905 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:28.968023 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.968133 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.968189 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:50:28.971745 17874 raft_consensus.cc:1261] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Refusing update from remote peer cd0cccee34ac47579460879f811ac657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:28.971971 18104 raft_consensus.cc:1261] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Refusing update from remote peer cd0cccee34ac47579460879f811ac657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:28.972296 18278 consensus_queue.cc:1035] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:28.972533 18224 consensus_queue.cc:1035] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:28.974155 18226 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: TxnStatusTablet state changed. Reason: New leader cd0cccee34ac47579460879f811ac657. Latest consensus state: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.974274 18226 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:28.975013 18223 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: TxnStatusTablet state changed. Reason: New leader cd0cccee34ac47579460879f811ac657. Latest consensus state: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.975157 18223 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:28.976680 18224 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.976804 18224 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:50:28.977149 18224 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.977238 18224 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P cd0cccee34ac47579460879f811ac657: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:50:28.979040 18226 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.979143 18226 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P 9b088df97c4a429c949ce6525d2c3346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:28.980877 18223 tablet_replica.cc:439] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd0cccee34ac47579460879f811ac657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 } } peers { permanent_uuid: "9b088df97c4a429c949ce6525d2c3346" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 45577 } } peers { permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 } } }
I20260808 04:50:28.980983 18223 tablet_replica.cc:442] T d3a96103e9c74710ab2c70deeb8c519d P b96c108fc91f4bebb55071a37a9cd280: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:28.986367 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.982581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.15.30.66:45577)
0808 04:50:28.982689 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.982693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.982910 (+ 217us) server_negotiation.cc:407] Connection header received
0808 04:50:28.982975 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.982977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.983024 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.983132 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:28.983371 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.983970 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.984871 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.985106 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.985213 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.985481 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:28.985902 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:28.986027 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.986109 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.986160 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:50:28.999317 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.994428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:37365 (local address 127.15.30.66:58833)
0808 04:50:28.994813 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:28.994826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:28.994918 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:28.995173 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:28.995175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:28.995183 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:28.995698 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.995714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.996804 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.996807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.997962 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:28.997973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.998592 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.998594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:28.998698 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.998742 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:28.999051 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:28.999056 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:50:28.999128 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:50:28.999193 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
I20260808 04:50:28.999321 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:28.994546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:58833 (local address 127.15.30.65:37365)
0808 04:50:28.994743 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:28.994748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:28.994909 (+ 161us) server_negotiation.cc:407] Connection header received
0808 04:50:28.995016 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:28.995020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:28.995078 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:28.995151 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:28.995838 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.996667 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.998102 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:28.998469 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:28.998576 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:28.998876 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:28.998959 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:28.999038 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:28.999143 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:50:28.999193 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 04:50:29.006681 18104 raft_consensus.cc:1261] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Refusing update from remote peer 9b088df97c4a429c949ce6525d2c3346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:29.007122 17659 raft_consensus.cc:1261] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Refusing update from remote peer 9b088df97c4a429c949ce6525d2c3346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:29.007268 18256 consensus_queue.cc:1035] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b96c108fc91f4bebb55071a37a9cd280" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:29.007524 18226 consensus_queue.cc:1035] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd0cccee34ac47579460879f811ac657" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:29.033059 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:29.028402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:40735 (local address 127.15.30.66:45577)
0808 04:50:29.028520 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:29.028525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:29.028540 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:29.028684 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:29.028687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:29.028734 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:29.028813 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:29.029371 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.030250 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.031611 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.031861 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.031969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:29.032564 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:29.032645 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:29.032684 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:29.032837 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 04:50:29.032906 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:50:29.033207 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:29.027950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:45577 (local address 127.15.30.65:40735)
0808 04:50:29.028474 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:29.028490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:29.028590 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:29.028931 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:29.028935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:29.029022 (+ 87us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:29.029244 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:29.029253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.030350 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.030353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:29.031497 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:29.031506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.032086 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.032094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:29.032297 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:29.032350 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:29.032737 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:29.032741 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:50:29.032819 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:50:29.032906 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":166,"threads_started":1}
I20260808 04:50:29.058233 17398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:29.058447 17398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:29.061589 17398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2177eab3b4cc4f39b8c411b36e4468f0: Sending DeleteTablet for 3 replicas of tablet 48acf885df0a42f69045b80e8b205296
I20260808 04:50:29.062233 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 48acf885df0a42f69045b80e8b205296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:29 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:50706
I20260808 04:50:29.062453 18248 tablet_replica.cc:330] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: stopping tablet replica
I20260808 04:50:29.062609 18248 raft_consensus.cc:2229] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:29.062824 18065 tablet_service.cc:1514] Processing DeleteTablet for tablet 48acf885df0a42f69045b80e8b205296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:29 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42952
I20260808 04:50:29.062935 18248 raft_consensus.cc:2258] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:29.063042 18247 tablet_replica.cc:330] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: stopping tablet replica
I20260808 04:50:29.063174 18247 raft_consensus.cc:2229] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:29.063326 18247 raft_consensus.cc:2258] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:29.064944 18248 ts_tablet_manager.cc:1905] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:29.068181 18248 ts_tablet_manager.cc:1918] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:29.068254 18248 log.cc:1196] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/48acf885df0a42f69045b80e8b205296
I20260808 04:50:29.069550 17622 tablet_service.cc:1514] Processing DeleteTablet for tablet 48acf885df0a42f69045b80e8b205296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:29 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:49712
I20260808 04:50:29.069752 18246 tablet_replica.cc:330] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: stopping tablet replica
I20260808 04:50:29.069892 18246 raft_consensus.cc:2229] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:29.070041 18246 raft_consensus.cc:2258] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:29.070077 18248 ts_tablet_manager.cc:1939] T 48acf885df0a42f69045b80e8b205296 P 9b088df97c4a429c949ce6525d2c3346: Deleting consensus metadata
I20260808 04:50:29.070923 18246 ts_tablet_manager.cc:1905] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:29.070909 17380 catalog_manager.cc:4841] TS 9b088df97c4a429c949ce6525d2c3346 (127.15.30.66:45577): tablet 48acf885df0a42f69045b80e8b205296 (table test-table [id=afe6ec7db0b74b3bbfd284923e475809]) successfully deleted
I20260808 04:50:29.072048 18247 ts_tablet_manager.cc:1905] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:29.073945 18246 ts_tablet_manager.cc:1918] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:29.074029 18246 log.cc:1196] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/48acf885df0a42f69045b80e8b205296
I20260808 04:50:29.074309 18246 ts_tablet_manager.cc:1939] T 48acf885df0a42f69045b80e8b205296 P cd0cccee34ac47579460879f811ac657: Deleting consensus metadata
I20260808 04:50:29.075742 17380 catalog_manager.cc:4841] TS cd0cccee34ac47579460879f811ac657 (127.15.30.65:37365): tablet 48acf885df0a42f69045b80e8b205296 (table test-table [id=afe6ec7db0b74b3bbfd284923e475809]) successfully deleted
I20260808 04:50:29.075438 18247 ts_tablet_manager.cc:1918] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:29.076012 18247 log.cc:1196] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/48acf885df0a42f69045b80e8b205296
I20260808 04:50:29.076283 18247 ts_tablet_manager.cc:1939] T 48acf885df0a42f69045b80e8b205296 P b96c108fc91f4bebb55071a37a9cd280: Deleting consensus metadata
Aug 08 04:50:29 dist-test-slave-zpvx krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:29.077471 17379 catalog_manager.cc:4841] TS b96c108fc91f4bebb55071a37a9cd280 (127.15.30.67:41285): tablet 48acf885df0a42f69045b80e8b205296 (table test-table [id=afe6ec7db0b74b3bbfd284923e475809]) successfully deleted
I20260808 04:50:29.082041 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:29.064564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.15.30.65:37365)
0808 04:50:29.069003 (+ 4439us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:29.069008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:29.069039 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:50:29.069084 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:29.069087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:29.069135 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:29.069270 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:29.070380 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.071213 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.072597 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.072942 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.073017 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:29.077553 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:29.077565 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:29.077582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:29.077610 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:29.079831 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:29.080414 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:29.080421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:29.080423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:29.080517 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:29.080874 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:29.080877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:29.080879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:29.081186 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:29.081378 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:29.081749 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 04:50:29.081877 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4350}
W20260808 04:50:29.083002 17709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49738
I20260808 04:50:29.094859 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:29.084940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.15.30.126:33333)
0808 04:50:29.085053 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:29.085058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:29.085520 (+ 462us) server_negotiation.cc:407] Connection header received
0808 04:50:29.085705 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:29.085708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:29.085761 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:29.085814 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:29.086726 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.087517 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.088866 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:29.089227 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:29.089401 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:29.091106 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:29.091110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:29.091112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:29.091139 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:29.092933 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:29.093507 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:29.093511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:29.093512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:29.093597 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:29.093886 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:29.093889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:29.093891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:29.094096 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:29.094196 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:29.094584 (+ 388us) server_negotiation.cc:299] Negotiation successful
0808 04:50:29.094701 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 04:50:29.095862 17398 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46402
I20260808 04:50:29.097441 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17515
I20260808 04:50:29.114898 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17748
I20260808 04:50:29.127126 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17963
I20260808 04:50:29.137019 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17353
2026-08-08T04:50:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2908 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:29 dist-test-slave-zpvx krb5kdc[18309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:29 dist-test-slave-zpvx krb5kdc[18309](info): set up 2 sockets
Aug 08 04:50:29 dist-test-slave-zpvx krb5kdc[18309](info): commencing operation
krb5kdc: starting...
W20260808 04:50:30.394230 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:30 dist-test-slave-zpvx krb5kdc[18309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:30.523299 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44067
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:44067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:30.637105 18329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:30.637420 18329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:30.637498 18329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:30.641502 18329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:30.641577 18329 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:50:30.641607 18329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:30.641664 18329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:30.641700 18329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:30.641736 18329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:30.648304 18329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:44067
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:30.649387 18329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:30.650440 18329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:30.658888 18335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:30.659073 18329 server_base.cc:1034] running on GCE node
W20260808 04:50:30.658888 18339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:30.659199 18336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:30.659839 18329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:30.660961 18329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:30.662142 18329 hybrid_clock.cc:648] HybridClock initialized: now 1786164630662141 us; error 43 us; skew 500 ppm
Aug 08 04:50:30 dist-test-slave-zpvx krb5kdc[18309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164630, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:30.669835 18329 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:30.670784 18329 webserver.cc:460] Webserver started at http://127.15.30.126:42665/ using document root <none> and password file <none>
I20260808 04:50:30.671365 18329 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:30.671473 18329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:30.671710 18329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:30.673834 18329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "73e8823ca278490bb5691d8aefc2a467"
format_stamp: "Formatted at 2026-08-08 04:50:30 on dist-test-slave-zpvx"
server_key: "8bcf798151ea40db884054abf29d5850"
server_key_iv: "8a760abe48012d9baee2f2b86cf6ef42"
server_key_version: "encryptionkey@0"
I20260808 04:50:30.674439 18329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "73e8823ca278490bb5691d8aefc2a467"
format_stamp: "Formatted at 2026-08-08 04:50:30 on dist-test-slave-zpvx"
server_key: "8bcf798151ea40db884054abf29d5850"
server_key_iv: "8a760abe48012d9baee2f2b86cf6ef42"
server_key_version: "encryptionkey@0"
I20260808 04:50:30.678220 18329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:50:30.680699 18351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:30.681916 18329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:30.682060 18329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "73e8823ca278490bb5691d8aefc2a467"
format_stamp: "Formatted at 2026-08-08 04:50:30 on dist-test-slave-zpvx"
server_key: "8bcf798151ea40db884054abf29d5850"
server_key_iv: "8a760abe48012d9baee2f2b86cf6ef42"
server_key_version: "encryptionkey@0"
I20260808 04:50:30.682173 18329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:30.735019 18329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:30.739609 18329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:30.739831 18329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:30.751278 18329 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:44067
I20260808 04:50:30.751300 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:44067 every 8 connection(s)
I20260808 04:50:30.752699 18329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:30.753422 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18329
I20260808 04:50:30.753548 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:30.753856 15481 external_mini_cluster.cc:1428] Setting key a1e553ab7bc06af1a26a7e81d8b7727a
I20260808 04:50:30.756137 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:30.762506 18446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Bootstrap starting.
Aug 08 04:50:30 dist-test-slave-zpvx krb5kdc[18309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:30.765270 18446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:30.766182 18446 log.cc:824] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:30.768146 18446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: No bootstrap required, opened a new log
I20260808 04:50:30.771497 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:30.755400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.15.30.126:44067)
0808 04:50:30.755918 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:30.755937 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:30.755979 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:50:30.756841 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:30.756863 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:30.757266 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:30.757580 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:30.759139 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:30.760643 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:30.762084 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:30.762788 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:30.762950 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:30.765909 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:30.765914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:30.765927 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:30.765956 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:30.768474 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:30.769023 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:30.769038 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:30.769043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:30.769175 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:30.769506 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:30.769508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:30.769510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:30.769925 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:30.770078 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:30.770412 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 04:50:30.770744 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":166,"threads_started":1}
I20260808 04:50:30.771888 18446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e8823ca278490bb5691d8aefc2a467" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44067 } }
I20260808 04:50:30.772142 18446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:30.772269 18446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e8823ca278490bb5691d8aefc2a467, State: Initialized, Role: FOLLOWER
I20260808 04:50:30.772713 18446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e8823ca278490bb5691d8aefc2a467" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44067 } }
I20260808 04:50:30.772867 18446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:30.772958 18446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:30.773077 18446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:30.774060 18446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e8823ca278490bb5691d8aefc2a467" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44067 } }
I20260808 04:50:30.774397 18446 leader_election.cc:302] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e8823ca278490bb5691d8aefc2a467; no voters:
I20260808 04:50:30.774715 18446 leader_election.cc:288] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:30.774838 18454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:30.775105 18454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [term 1 LEADER]: Becoming Leader. State: Replica: 73e8823ca278490bb5691d8aefc2a467, State: Running, Role: LEADER
I20260808 04:50:30.775400 18454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e8823ca278490bb5691d8aefc2a467" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44067 } }
I20260808 04:50:30.775929 18446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:30.777210 18458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e8823ca278490bb5691d8aefc2a467. Latest consensus state: current_term: 1 leader_uuid: "73e8823ca278490bb5691d8aefc2a467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e8823ca278490bb5691d8aefc2a467" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44067 } } }
I20260808 04:50:30.777350 18458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:30.777637 18457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73e8823ca278490bb5691d8aefc2a467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e8823ca278490bb5691d8aefc2a467" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44067 } } }
I20260808 04:50:30.777730 18457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:30.777891 18469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:30.780587 18469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:30.786056 18469 catalog_manager.cc:1347] Generated new cluster ID: 8f20deb5d3c745488605120d28f8f5f4
I20260808 04:50:30.786119 18469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:30.807252 18469 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:30.808527 18469 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:30.818475 18469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Generated new TSK 0
I20260808 04:50:30.819238 18469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:30.859261 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44067
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:30.972002 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:30.972244 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:30.972288 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:30.976233 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:30.976286 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:30.976369 18485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:50:30.976435 18485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:50:30.976497 18485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:30.981546 18485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:30.982436 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:30.983341 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:30.992734 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:30.992739 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:30.992739 18493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:30.993467 18485 server_base.cc:1034] running on GCE node
I20260808 04:50:30.993818 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:30.994421 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:30.995581 18485 hybrid_clock.cc:648] HybridClock initialized: now 1786164630995567 us; error 25 us; skew 500 ppm
Aug 08 04:50:30 dist-test-slave-zpvx krb5kdc[18309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164630, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:30.998291 18485 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:30.999153 18485 webserver.cc:460] Webserver started at http://127.15.30.65:44357/ using document root <none> and password file <none>
I20260808 04:50:30.999706 18485 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:30.999796 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:31.000026 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:31.002094 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "2739ca8d72d44e4db69a4ee62fc45e9e"
format_stamp: "Formatted at 2026-08-08 04:50:30 on dist-test-slave-zpvx"
server_key: "5204c97c6716a0db2958037e4656610a"
server_key_iv: "9c94ab4a62b98725132a74d8cec7466c"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.002660 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "2739ca8d72d44e4db69a4ee62fc45e9e"
format_stamp: "Formatted at 2026-08-08 04:50:30 on dist-test-slave-zpvx"
server_key: "5204c97c6716a0db2958037e4656610a"
server_key_iv: "9c94ab4a62b98725132a74d8cec7466c"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.006459 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:50:31.008824 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:31.010054 18485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:31.010201 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "2739ca8d72d44e4db69a4ee62fc45e9e"
format_stamp: "Formatted at 2026-08-08 04:50:30 on dist-test-slave-zpvx"
server_key: "5204c97c6716a0db2958037e4656610a"
server_key_iv: "9c94ab4a62b98725132a74d8cec7466c"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.010314 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:31.049649 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:31.053208 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:31.053417 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:31.054896 18485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:31.054970 18485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:31.055037 18485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:31.055068 18485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164630, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:31.066249 18485 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:36275
I20260808 04:50:31.066311 18675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:36275 every 8 connection(s)
I20260808 04:50:31.067157 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:31.070343 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.055997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:44605 (local address 127.15.30.126:44067)
0808 04:50:31.056145 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.056157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.057135 (+ 978us) server_negotiation.cc:407] Connection header received
0808 04:50:31.058486 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.058490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.058557 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.058661 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.060454 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.061383 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.063274 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.063510 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.063592 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.066691 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.066695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.066697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.066726 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.068420 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.068919 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.068922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.068924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.069002 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.069338 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.069341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.069342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.069526 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.069602 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.070071 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.070210 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:50:31.071045 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.056420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44067 (local address 127.15.30.65:44605)
0808 04:50:31.056948 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.057008 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.058223 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.058896 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.058910 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.059443 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:31.060208 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.060234 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.061575 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.061581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.063149 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.063160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.063686 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.063691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.063909 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.064633 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:31.064667 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:31.066520 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:31.068536 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.068541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.068552 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.068812 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.069097 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.069099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.069101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.069242 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.069616 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:31.069621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:31.069843 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.069989 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.070234 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260808 04:50:31.072559 18676 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44067
I20260808 04:50:31.072737 18676 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:31.073282 18676 heartbeater.cc:511] Master 127.15.30.126:44067 requested a full tablet report, sending...
I20260808 04:50:31.075166 18376 ts_manager.cc:194] Registered new tserver with Master: 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275)
I20260808 04:50:31.076460 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18485
I20260808 04:50:31.076587 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:31.076699 18376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:44605
I20260808 04:50:31.076932 15481 external_mini_cluster.cc:1428] Setting key 782ee3564d3c8af1037229546c7c4b20
I20260808 04:50:31.082604 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.074627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44067 (local address 127.15.30.65:59391)
0808 04:50:31.075043 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.075058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.075171 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.075456 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.075460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.075679 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:31.075787 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.075793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.076757 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.076760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.077497 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.077507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.078038 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.078042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.078132 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.078656 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:31.078670 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:31.079090 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:31.081057 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.081062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.081065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.081377 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.081648 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.081653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.081656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.081756 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.082097 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:31.082101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:31.082219 (+ 118us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.082355 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.082458 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":107,"threads_started":1}
I20260808 04:50:31.082795 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.074772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:59391 (local address 127.15.30.126:44067)
0808 04:50:31.074884 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.074889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.075140 (+ 251us) server_negotiation.cc:407] Connection header received
0808 04:50:31.075261 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.075263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.075343 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.075436 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.075907 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.076510 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.077609 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.077911 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.078015 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.079252 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.079259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.079265 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.079297 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.080946 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.081481 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.081484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.081485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.081557 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.081849 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.081851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.081852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.082001 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.082075 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.082440 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.082574 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:31.107503 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44067
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:31.218664 18695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:31.218891 18695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:31.218962 18695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:31.222957 18695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:31.223011 18695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:31.223085 18695 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:50:31.223119 18695 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:50:31.223186 18695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:31.228190 18695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:31.229171 18695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:31.230139 18695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:50:31.239485 18695 server_base.cc:1034] running on GCE node
W20260808 04:50:31.239413 18702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:31.239413 18705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:31.239406 18703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:31.240137 18695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:31.241042 18695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:31.242210 18695 hybrid_clock.cc:648] HybridClock initialized: now 1786164631242196 us; error 25 us; skew 500 ppm
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164631, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:31.245056 18695 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:31.245831 18695 webserver.cc:460] Webserver started at http://127.15.30.66:39881/ using document root <none> and password file <none>
I20260808 04:50:31.246335 18695 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:31.246392 18695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:31.246552 18695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:31.248551 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "2976192c711f45768457228b28546c86"
format_stamp: "Formatted at 2026-08-08 04:50:31 on dist-test-slave-zpvx"
server_key: "a6bb379e807e377a4f1ce74d0569ae31"
server_key_iv: "0bc6aa8975d4f34be8b4887acf704a5d"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.249101 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "2976192c711f45768457228b28546c86"
format_stamp: "Formatted at 2026-08-08 04:50:31 on dist-test-slave-zpvx"
server_key: "a6bb379e807e377a4f1ce74d0569ae31"
server_key_iv: "0bc6aa8975d4f34be8b4887acf704a5d"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.252753 18695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:50:31.255009 18714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:31.256173 18695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:31.256292 18695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "2976192c711f45768457228b28546c86"
format_stamp: "Formatted at 2026-08-08 04:50:31 on dist-test-slave-zpvx"
server_key: "a6bb379e807e377a4f1ce74d0569ae31"
server_key_iv: "0bc6aa8975d4f34be8b4887acf704a5d"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.256443 18695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:31.279649 18695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:31.282684 18695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:31.282835 18695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:31.284135 18695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:31.284185 18695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:31.284225 18695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:31.284293 18695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164631, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:31.294934 18695 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:39933
I20260808 04:50:31.295053 18899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:39933 every 8 connection(s)
I20260808 04:50:31.296057 18695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:31.298961 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.285052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:50323 (local address 127.15.30.126:44067)
0808 04:50:31.285205 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.285209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.286370 (+ 1161us) server_negotiation.cc:407] Connection header received
0808 04:50:31.287639 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.287645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.287721 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.287821 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.289445 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.290319 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.291877 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.292241 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.292409 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.295514 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.295518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.295521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.295551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.297240 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.297713 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.297715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.297716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.297789 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.298045 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.298047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.298048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.298238 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.298312 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.298743 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.298841 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:50:31.299747 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.285608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44067 (local address 127.15.30.66:50323)
0808 04:50:31.286211 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.286271 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.287334 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.288047 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.288058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.288560 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:31.289295 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.289315 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.290473 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.290477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.291723 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.291735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.292476 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.292480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.292629 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.293380 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:31.293429 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:31.295339 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:31.297343 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.297348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.297364 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.297620 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.297848 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.297850 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.297851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.297969 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.298327 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:31.298331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:31.298541 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.298694 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.298931 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":136,"threads_started":1}
I20260808 04:50:31.300964 18900 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44067
I20260808 04:50:31.301120 18900 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:31.301579 18900 heartbeater.cc:511] Master 127.15.30.126:44067 requested a full tablet report, sending...
I20260808 04:50:31.302708 18362 ts_manager.cc:194] Registered new tserver with Master: 2976192c711f45768457228b28546c86 (127.15.30.66:39933)
I20260808 04:50:31.303436 18362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:50323
I20260808 04:50:31.303931 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18695
I20260808 04:50:31.304047 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:31.304414 15481 external_mini_cluster.cc:1428] Setting key 8c911db4aa541d506536cd672f43841b
I20260808 04:50:31.311157 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.303274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:47515 (local address 127.15.30.126:44067)
0808 04:50:31.303394 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.303398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.303580 (+ 182us) server_negotiation.cc:407] Connection header received
0808 04:50:31.303686 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.303689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.303737 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.303819 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.304445 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.305187 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.306600 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.306992 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.307118 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.308183 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.308186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.308188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.308208 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.309503 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.309946 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.309950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.309952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.310037 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.310334 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.310338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.310340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.310541 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.310659 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.310914 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.311020 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:50:31.311188 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.303208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44067 (local address 127.15.30.66:47515)
0808 04:50:31.303457 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.303471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.303592 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.303936 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.303939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.304164 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:31.304293 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.304300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.305327 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.305330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.306471 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.306485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.307117 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.307119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.307185 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.307659 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:31.307678 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:31.308028 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:31.309625 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.309629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.309631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.309833 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.310136 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.310140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.310142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.310238 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.310642 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:31.310645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:31.310752 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.310861 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.311002 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:31.333510 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44067
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:31.446916 18912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:31.447172 18912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:31.447255 18912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:31.451329 18912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:31.451388 18912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:31.451426 18912 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:50:31.451479 18912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:50:31.451534 18912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:31.456732 18912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:31.457784 18912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:31.458871 18912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:31.468789 18917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:31.468789 18919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:31.469067 18912 server_base.cc:1034] running on GCE node
W20260808 04:50:31.468869 18922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:31.469662 18912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:31.470243 18912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:31.471436 18912 hybrid_clock.cc:648] HybridClock initialized: now 1786164631471413 us; error 32 us; skew 500 ppm
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164631, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:31.474552 18912 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:31.475492 18912 webserver.cc:460] Webserver started at http://127.15.30.67:37357/ using document root <none> and password file <none>
I20260808 04:50:31.476111 18912 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:31.476176 18912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:31.476460 18912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:31.478665 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "8362b15555084bff89c57785ead867da"
format_stamp: "Formatted at 2026-08-08 04:50:31 on dist-test-slave-zpvx"
server_key: "10245997a5c83229cdd5ea01f2c46446"
server_key_iv: "2a0c22c96fec0e5d4353970edb8aeb03"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.479279 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "8362b15555084bff89c57785ead867da"
format_stamp: "Formatted at 2026-08-08 04:50:31 on dist-test-slave-zpvx"
server_key: "10245997a5c83229cdd5ea01f2c46446"
server_key_iv: "2a0c22c96fec0e5d4353970edb8aeb03"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.483342 18912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260808 04:50:31.485966 18934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:31.487319 18912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:31.487495 18912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "8362b15555084bff89c57785ead867da"
format_stamp: "Formatted at 2026-08-08 04:50:31 on dist-test-slave-zpvx"
server_key: "10245997a5c83229cdd5ea01f2c46446"
server_key_iv: "2a0c22c96fec0e5d4353970edb8aeb03"
server_key_version: "encryptionkey@0"
I20260808 04:50:31.487627 18912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:31.507623 18912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:31.511025 18912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:31.511276 18912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:31.513024 18912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:31.513108 18912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:31.513185 18912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:31.513232 18912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164631, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:31.525341 18912 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:37575
I20260808 04:50:31.525381 19129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:37575 every 8 connection(s)
I20260808 04:50:31.526628 18912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:31.529805 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.513806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:44189 (local address 127.15.30.126:44067)
0808 04:50:31.513960 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.513967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.515699 (+ 1732us) server_negotiation.cc:407] Connection header received
0808 04:50:31.517189 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.517196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.517270 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.517372 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.518915 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.519816 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.521880 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.522103 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.522191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.525599 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.525604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.525607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.525637 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.527478 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.528105 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.528109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.528111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.528207 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.528628 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.528631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.528633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.528831 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.528952 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.529482 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.529610 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:50:31.530052 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18912
I20260808 04:50:31.530184 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:31.530474 15481 external_mini_cluster.cc:1428] Setting key 3a0e73bd8fe21803e7ffc02bd8ee4e6c
I20260808 04:50:31.530721 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.514968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44067 (local address 127.15.30.67:44189)
0808 04:50:31.515542 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.515588 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.516900 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.517530 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.517536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.518075 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:31.518767 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.518789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.519992 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.520000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.521752 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.521763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.522286 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.522292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.522547 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.523290 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:31.523318 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:31.525362 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:31.527599 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.527605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.527619 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.527978 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.528312 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.528317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.528319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.528507 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.528948 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:31.528955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:31.529220 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.529411 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.529745 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":150,"threads_started":1}
I20260808 04:50:31.532033 19130 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44067
I20260808 04:50:31.532191 19130 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:31.532862 19130 heartbeater.cc:511] Master 127.15.30.126:44067 requested a full tablet report, sending...
I20260808 04:50:31.533915 18362 ts_manager.cc:194] Registered new tserver with Master: 8362b15555084bff89c57785ead867da (127.15.30.67:37575)
I20260808 04:50:31.534639 18362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:44189
I20260808 04:50:31.541599 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.534378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:53263 (local address 127.15.30.126:44067)
0808 04:50:31.534513 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.534518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.534676 (+ 158us) server_negotiation.cc:407] Connection header received
0808 04:50:31.534835 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.534839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.534895 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.534981 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.535474 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.536322 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.537149 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.537375 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.537461 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.538473 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.538476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.538478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.538501 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.540015 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.540540 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.540543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.540544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.540622 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.540935 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.540937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.540938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.541090 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.541175 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.541412 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.541499 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:50:31.541615 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.534313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44067 (local address 127.15.30.67:53263)
0808 04:50:31.534594 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.534609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.534722 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.535016 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.535019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.535241 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:31.535354 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.535360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.536491 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.536494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.537039 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.537047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.537479 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.537481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.537550 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.537983 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:31.537994 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:31.538340 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:31.540171 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.540174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.540175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.540411 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.540749 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.540752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.540753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.540843 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.541182 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:31.541185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:31.541262 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.541354 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.541460 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":123,"threads_started":1}
I20260808 04:50:31.544529 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:31.570529 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.562788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.15.30.126:44067)
0808 04:50:31.562914 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.562917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.562929 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:50:31.562964 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.562966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.563001 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.563066 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.563714 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.564202 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.565060 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.565264 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.565362 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.567604 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.567607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.567609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.567631 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.569116 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.569480 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.569483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.569484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.569554 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.569827 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.569829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.569830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.569977 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.570077 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.570247 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.570334 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:50:31.573271 18362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:31.576481 18362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:31.591774 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.585209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.15.30.67:37575)
0808 04:50:31.585576 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.585590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.586125 (+ 535us) server_negotiation.cc:407] Connection header received
0808 04:50:31.586213 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.586219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.586608 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.586777 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:31.587281 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.588549 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.590463 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.591051 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.591172 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.591458 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.591514 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.591600 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":150,"threads_started":1}
I20260808 04:50:31.591799 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.585177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:39933 (local address 127.0.0.1:39462)
0808 04:50:31.586414 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.586430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.586523 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.587149 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.587151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.587167 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:31.587273 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.587279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.588711 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.588714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.590047 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.590059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.591360 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.591363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.591472 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.591495 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.591576 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.591653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":57,"thread_start_us":99,"threads_started":1}
I20260808 04:50:31.591837 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.585049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37575 (local address 127.0.0.1:44262)
0808 04:50:31.585759 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.585811 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.585949 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.586954 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.586966 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.586988 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:31.587144 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.587151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.588666 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.588669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.590320 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.590334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.591191 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.591195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.591327 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.591369 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.591476 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.591598 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":253,"thread_start_us":92,"threads_started":1}
I20260808 04:50:31.592283 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.585801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:36275 (local address 127.0.0.1:42294)
0808 04:50:31.586709 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.586723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.586809 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.587342 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.587345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.587364 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:31.587470 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.587477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.588540 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.588546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.590360 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.590371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.591662 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.591666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.591746 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.591767 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.592042 (+ 275us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.592122 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":250,"thread_start_us":87,"threads_started":1}
I20260808 04:50:31.592829 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.585874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.15.30.66:39933)
0808 04:50:31.586567 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.586581 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.586603 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:50:31.586784 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.586791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.586979 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.587124 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:31.587403 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.588608 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.590176 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.591010 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.591124 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.591290 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.592590 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.592693 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":128,"threads_started":1}
I20260808 04:50:31.592849 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.585982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.15.30.65:36275)
0808 04:50:31.586350 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.586355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.586896 (+ 541us) server_negotiation.cc:407] Connection header received
0808 04:50:31.586974 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.586978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.587182 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.587364 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:31.587609 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.588431 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.590501 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.591375 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.591472 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.591605 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.592640 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.592723 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":69,"threads_started":1}
I20260808 04:50:31.595088 18577 tablet_service.cc:1467] Processing CreateTablet for tablet eaa4a0b970334e889c0e8c816f1de8b2 (DEFAULT_TABLE table=test-table [id=526211ce4b54460cb8ae4f89f43ce1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:31.595022 19017 tablet_service.cc:1467] Processing CreateTablet for tablet eaa4a0b970334e889c0e8c816f1de8b2 (DEFAULT_TABLE table=test-table [id=526211ce4b54460cb8ae4f89f43ce1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:31.595389 18792 tablet_service.cc:1467] Processing CreateTablet for tablet eaa4a0b970334e889c0e8c816f1de8b2 (DEFAULT_TABLE table=test-table [id=526211ce4b54460cb8ae4f89f43ce1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:31.596071 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa4a0b970334e889c0e8c816f1de8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.596095 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa4a0b970334e889c0e8c816f1de8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.596259 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa4a0b970334e889c0e8c816f1de8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.602437 19161 tablet_bootstrap.cc:492] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Bootstrap starting.
I20260808 04:50:31.603673 19163 tablet_bootstrap.cc:492] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Bootstrap starting.
I20260808 04:50:31.604558 19164 tablet_bootstrap.cc:492] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Bootstrap starting.
I20260808 04:50:31.604935 19161 tablet_bootstrap.cc:654] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.605775 19163 tablet_bootstrap.cc:654] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.605831 19161 log.cc:824] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:31.606349 19164 tablet_bootstrap.cc:654] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.606544 19163 log.cc:824] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:31.607142 19164 log.cc:824] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:31.607993 19161 tablet_bootstrap.cc:492] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: No bootstrap required, opened a new log
I20260808 04:50:31.608201 19161 ts_tablet_manager.cc:1397] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:50:31.608664 19163 tablet_bootstrap.cc:492] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: No bootstrap required, opened a new log
I20260808 04:50:31.608867 19163 ts_tablet_manager.cc:1397] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260808 04:50:31.609236 19164 tablet_bootstrap.cc:492] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: No bootstrap required, opened a new log
I20260808 04:50:31.609437 19164 ts_tablet_manager.cc:1397] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:50:31.611712 19163 raft_consensus.cc:348] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.611979 19163 raft_consensus.cc:374] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.612102 19163 raft_consensus.cc:729] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8362b15555084bff89c57785ead867da, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.612625 19163 consensus_queue.cc:260] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.612934 19161 raft_consensus.cc:348] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.613229 19161 raft_consensus.cc:374] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.613267 19130 heartbeater.cc:503] Master 127.15.30.126:44067 was elected leader, sending a full tablet report...
I20260808 04:50:31.613298 19161 raft_consensus.cc:729] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2739ca8d72d44e4db69a4ee62fc45e9e, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.613492 19164 raft_consensus.cc:348] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.613716 19164 raft_consensus.cc:374] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.613780 19164 raft_consensus.cc:729] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2976192c711f45768457228b28546c86, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.613755 19161 consensus_queue.cc:260] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.614406 19164 consensus_queue.cc:260] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.614764 19161 ts_tablet_manager.cc:1428] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260808 04:50:31.614925 19163 ts_tablet_manager.cc:1428] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:50:31.615077 18676 heartbeater.cc:503] Master 127.15.30.126:44067 was elected leader, sending a full tablet report...
I20260808 04:50:31.615453 18900 heartbeater.cc:503] Master 127.15.30.126:44067 was elected leader, sending a full tablet report...
I20260808 04:50:31.616231 19164 ts_tablet_manager.cc:1428] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:50:31.629810 19171 raft_consensus.cc:482] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:31.629940 19171 raft_consensus.cc:504] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.630848 19171 leader_election.cc:288] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2976192c711f45768457228b28546c86 (127.15.30.66:39933), 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275)
I20260808 04:50:31.635048 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.631073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:39933 (local address 127.15.30.67:51789)
0808 04:50:31.631231 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.631247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.631337 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.631641 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.631647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.631670 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:31.631822 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.631827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.632823 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.632827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.634051 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.634061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.634754 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.634756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.634830 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.634849 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.634901 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.634954 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"spinlock_wait_cycles":6144}
I20260808 04:50:31.635100 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.631303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:51789 (local address 127.15.30.66:39933)
0808 04:50:31.631418 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.631422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.631436 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:50:31.631481 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.631487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.631537 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.631609 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:31.631950 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.632709 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.634182 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.634637 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.634760 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.634815 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.634923 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.634985 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:50:31.635691 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.631073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:36275 (local address 127.15.30.67:37017)
0808 04:50:31.631574 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.631606 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.631701 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.631910 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.631913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.631933 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:31.632548 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.632560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.633502 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.633505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.634471 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.634479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.635336 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.635339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.635434 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.635456 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.635536 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.635595 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":144,"spinlock_wait_cycles":384,"thread_start_us":130,"threads_started":1}
I20260808 04:50:31.635746 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.631169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:37017 (local address 127.15.30.65:36275)
0808 04:50:31.631301 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.631306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.631704 (+ 398us) server_negotiation.cc:407] Connection header received
0808 04:50:31.631779 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.631781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.631816 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.631874 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:31.632685 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.633404 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.634594 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.635221 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.635358 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.635407 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.635569 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.635639 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:50:31.635779 18827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4a0b970334e889c0e8c816f1de8b2" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2976192c711f45768457228b28546c86" is_pre_election: true
I20260808 04:50:31.636065 18827 raft_consensus.cc:2454] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 0.
I20260808 04:50:31.636456 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4a0b970334e889c0e8c816f1de8b2" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" is_pre_election: true
I20260808 04:50:31.636561 18935 leader_election.cc:302] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2976192c711f45768457228b28546c86, 8362b15555084bff89c57785ead867da; no voters:
I20260808 04:50:31.636734 18588 raft_consensus.cc:2454] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 0.
I20260808 04:50:31.636858 19171 raft_consensus.cc:2781] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:31.636958 19171 raft_consensus.cc:482] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:31.637018 19171 raft_consensus.cc:3037] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:31.638007 19171 raft_consensus.cc:504] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.638296 19171 leader_election.cc:288] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 election: Requested vote from peers 2976192c711f45768457228b28546c86 (127.15.30.66:39933), 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275)
I20260808 04:50:31.638623 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4a0b970334e889c0e8c816f1de8b2" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e"
I20260808 04:50:31.638620 18827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4a0b970334e889c0e8c816f1de8b2" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2976192c711f45768457228b28546c86"
I20260808 04:50:31.638717 18588 raft_consensus.cc:3037] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:31.638753 18827 raft_consensus.cc:3037] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:31.639698 18588 raft_consensus.cc:2454] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 1.
I20260808 04:50:31.639806 18827 raft_consensus.cc:2454] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 1.
I20260808 04:50:31.640051 18939 leader_election.cc:302] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2739ca8d72d44e4db69a4ee62fc45e9e, 8362b15555084bff89c57785ead867da; no voters:
I20260808 04:50:31.640285 19171 raft_consensus.cc:2781] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:31.640524 19171 raft_consensus.cc:686] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 1 LEADER]: Becoming Leader. State: Replica: 8362b15555084bff89c57785ead867da, State: Running, Role: LEADER
I20260808 04:50:31.640816 19171 consensus_queue.cc:237] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } }
I20260808 04:50:31.643862 18362 catalog_manager.cc:5495] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da reported cstate change: term changed from 0 to 1, leader changed from <none> to 8362b15555084bff89c57785ead867da (127.15.30.67). New cstate: current_term: 1 leader_uuid: "8362b15555084bff89c57785ead867da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } health_report { overall_health: UNKNOWN } } }
Aug 08 04:50:31 dist-test-slave-zpvx krb5kdc[18309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164630, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:31.694690 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.686064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.15.30.126:44067)
0808 04:50:31.686197 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.686204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.686370 (+ 166us) server_negotiation.cc:407] Connection header received
0808 04:50:31.686516 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.686520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.686564 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.686674 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:31.687404 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.687927 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.688785 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.688996 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.689116 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.691429 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:31.691432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:31.691434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:31.691457 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:31.693025 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.693527 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.693530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.693531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.693612 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:31.693861 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:31.693864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:31.693865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:31.694029 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:31.694139 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.694452 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.694564 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":44}
I20260808 04:50:31.694692 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.686024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44067 (local address 127.0.0.1:59964)
0808 04:50:31.686284 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.686296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.686425 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.686691 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.686693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.687174 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:31.687290 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.687297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.688034 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.688040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.688668 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.688677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.689124 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.689127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.689208 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.689815 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:31.689827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:31.691295 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:31.693168 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.693178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.693183 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.693434 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.693696 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:31.693698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:31.693699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:31.693780 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:31.694185 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:31.694191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:31.694293 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.694395 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.694512 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
I20260808 04:50:31.698357 18362 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:59964:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:50:31.699055 18362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:31.704087 18577 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed3ae118f214fa19790f298518b9d99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=796dc592d41a4117b39a887bd275d14f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:50:31.704375 18792 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed3ae118f214fa19790f298518b9d99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=796dc592d41a4117b39a887bd275d14f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:50:31.704882 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed3ae118f214fa19790f298518b9d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.704473 19017 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed3ae118f214fa19790f298518b9d99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=796dc592d41a4117b39a887bd275d14f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:50:31.705103 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed3ae118f214fa19790f298518b9d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.705135 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed3ae118f214fa19790f298518b9d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.707810 19164 tablet_bootstrap.cc:492] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: Bootstrap starting.
I20260808 04:50:31.708143 19163 tablet_bootstrap.cc:492] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: Bootstrap starting.
I20260808 04:50:31.708182 19161 tablet_bootstrap.cc:492] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: Bootstrap starting.
I20260808 04:50:31.708936 19164 tablet_bootstrap.cc:654] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.709131 19163 tablet_bootstrap.cc:654] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.709364 19161 tablet_bootstrap.cc:654] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.710693 19163 tablet_bootstrap.cc:492] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: No bootstrap required, opened a new log
I20260808 04:50:31.710731 19164 tablet_bootstrap.cc:492] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: No bootstrap required, opened a new log
I20260808 04:50:31.710804 19163 ts_tablet_manager.cc:1397] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:31.710815 19164 ts_tablet_manager.cc:1397] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:31.711068 19161 tablet_bootstrap.cc:492] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: No bootstrap required, opened a new log
I20260808 04:50:31.711160 19161 ts_tablet_manager.cc:1397] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:31.711280 19163 raft_consensus.cc:348] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.711304 19164 raft_consensus.cc:348] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.711395 19163 raft_consensus.cc:374] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.711406 19164 raft_consensus.cc:374] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.711439 19164 raft_consensus.cc:729] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2976192c711f45768457228b28546c86, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.711436 19163 raft_consensus.cc:729] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8362b15555084bff89c57785ead867da, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.711582 19163 consensus_queue.cc:260] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.711571 19164 consensus_queue.cc:260] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.711644 19161 raft_consensus.cc:348] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.711755 19161 raft_consensus.cc:374] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.711889 19163 ts_tablet_manager.cc:1428] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:31.712096 19171 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.712263 19171 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:31.711887 19173 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.712589 19173 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:31.712816 19164 ts_tablet_manager.cc:1428] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:31.711798 19161 raft_consensus.cc:729] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2739ca8d72d44e4db69a4ee62fc45e9e, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.713528 19161 consensus_queue.cc:260] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.713835 19161 ts_tablet_manager.cc:1428] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260808 04:50:31.713855 19172 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.714013 19172 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:31.776990 19172 raft_consensus.cc:482] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:31.777177 19172 raft_consensus.cc:504] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
W20260808 04:50:31.777788 19131 tablet.cc:2369] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:31.778481 19172 leader_election.cc:288] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2976192c711f45768457228b28546c86 (127.15.30.66:39933), 8362b15555084bff89c57785ead867da (127.15.30.67:37575)
I20260808 04:50:31.782884 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.778871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:45165 (local address 127.15.30.67:37575)
0808 04:50:31.779013 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.779018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.779286 (+ 268us) server_negotiation.cc:407] Connection header received
0808 04:50:31.779476 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.779479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.779537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.779597 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:31.779959 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.780789 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.781879 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.782289 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.782433 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.782492 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.782686 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.782767 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:50:31.782910 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.778748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37575 (local address 127.15.30.65:45165)
0808 04:50:31.779182 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.779194 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.779329 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.779657 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.779660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.779676 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:31.779834 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.779841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.780891 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.780897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.781744 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.781754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.782436 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.782440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.782556 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.782575 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.782667 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.782737 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":27,"thread_start_us":166,"threads_started":1}
I20260808 04:50:31.783407 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.778873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:58085 (local address 127.15.30.66:39933)
0808 04:50:31.779014 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.779018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.779360 (+ 342us) server_negotiation.cc:407] Connection header received
0808 04:50:31.779522 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.779524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.779566 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.779661 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:31.780596 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.781418 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.782412 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.782813 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.782943 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.782989 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.783197 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.783275 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:50:31.783542 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed3ae118f214fa19790f298518b9d99" candidate_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8362b15555084bff89c57785ead867da" is_pre_election: true
I20260808 04:50:31.783797 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.778749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:39933 (local address 127.15.30.65:58085)
0808 04:50:31.779275 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.779287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.779408 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.779694 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.779700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.779722 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:31.780461 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.780473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.781565 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.781569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.782279 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.782288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.782939 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.782943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.783049 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.783070 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.783560 (+ 490us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.783644 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":60,"thread_start_us":124,"threads_started":1}
I20260808 04:50:31.783793 19048 raft_consensus.cc:2454] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2739ca8d72d44e4db69a4ee62fc45e9e in term 0.
I20260808 04:50:31.784391 18827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed3ae118f214fa19790f298518b9d99" candidate_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2976192c711f45768457228b28546c86" is_pre_election: true
I20260808 04:50:31.784406 18504 leader_election.cc:302] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2739ca8d72d44e4db69a4ee62fc45e9e, 8362b15555084bff89c57785ead867da; no voters:
I20260808 04:50:31.784512 18827 raft_consensus.cc:2454] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2739ca8d72d44e4db69a4ee62fc45e9e in term 0.
I20260808 04:50:31.784693 19172 raft_consensus.cc:2781] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:31.784768 19172 raft_consensus.cc:482] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:31.784811 19172 raft_consensus.cc:3037] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:31.785837 19172 raft_consensus.cc:504] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.786244 19172 leader_election.cc:288] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [CANDIDATE]: Term 1 election: Requested vote from peers 2976192c711f45768457228b28546c86 (127.15.30.66:39933), 8362b15555084bff89c57785ead867da (127.15.30.67:37575)
I20260808 04:50:31.786549 18827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed3ae118f214fa19790f298518b9d99" candidate_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2976192c711f45768457228b28546c86"
I20260808 04:50:31.786680 18827 raft_consensus.cc:3037] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:31.786686 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed3ae118f214fa19790f298518b9d99" candidate_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8362b15555084bff89c57785ead867da"
I20260808 04:50:31.786805 19048 raft_consensus.cc:3037] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:31.787900 18827 raft_consensus.cc:2454] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2739ca8d72d44e4db69a4ee62fc45e9e in term 1.
I20260808 04:50:31.788043 19048 raft_consensus.cc:2454] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2739ca8d72d44e4db69a4ee62fc45e9e in term 1.
I20260808 04:50:31.788306 18502 leader_election.cc:302] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2739ca8d72d44e4db69a4ee62fc45e9e, 2976192c711f45768457228b28546c86; no voters:
I20260808 04:50:31.788549 19172 raft_consensus.cc:2781] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:31.788766 19172 raft_consensus.cc:686] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 LEADER]: Becoming Leader. State: Replica: 2739ca8d72d44e4db69a4ee62fc45e9e, State: Running, Role: LEADER
I20260808 04:50:31.789103 19172 consensus_queue.cc:237] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.791159 19206 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: TxnStatusTablet state changed. Reason: New leader 2739ca8d72d44e4db69a4ee62fc45e9e. Latest consensus state: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.791276 19206 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:50:31.791546 19210 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:50:31.791848 19210 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:50:31.792464 18373 catalog_manager.cc:5495] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65). New cstate: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } health_report { overall_health: UNKNOWN } } }
W20260808 04:50:31.797534 18902 tablet.cc:2369] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:31.810021 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.806038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:36275 (local address 127.0.0.1:42310)
0808 04:50:31.806198 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.806217 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.806305 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.806630 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.806632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.806637 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:31.806725 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.806729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.807698 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.807701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.808364 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.808421 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.808978 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.808980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.809057 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.809099 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:31.809797 (+ 698us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:31.809808 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.809858 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.809918 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 04:50:31.810062 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.806123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.15.30.65:36275)
0808 04:50:31.806235 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.806240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.806258 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:50:31.806447 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.806450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.806510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.806615 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:31.806837 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.807586 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.808530 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.808876 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.808978 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.809237 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:31.809691 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:31.809814 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.809872 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.809923 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:50:31.815423 19048 raft_consensus.cc:1261] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Refusing update from remote peer 2739ca8d72d44e4db69a4ee62fc45e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:31.815503 18827 raft_consensus.cc:1261] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Refusing update from remote peer 2739ca8d72d44e4db69a4ee62fc45e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:31.816131 19206 consensus_queue.cc:1035] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:31.816332 19172 consensus_queue.cc:1035] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:50:31.819276 18678 tablet.cc:2369] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:31.820050 19171 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: TxnStatusTablet state changed. Reason: New leader 2739ca8d72d44e4db69a4ee62fc45e9e. Latest consensus state: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.820165 19171 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:31.820318 19173 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: TxnStatusTablet state changed. Reason: New leader 2739ca8d72d44e4db69a4ee62fc45e9e. Latest consensus state: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.820446 19173 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:31.823580 19206 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.823695 19206 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:50:31.823836 19206 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.823927 19206 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 2739ca8d72d44e4db69a4ee62fc45e9e: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:50:31.824208 19171 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.824306 19171 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 8362b15555084bff89c57785ead867da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:31.825599 19173 tablet_replica.cc:439] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } } }
I20260808 04:50:31.825707 19173 tablet_replica.cc:442] T 2ed3ae118f214fa19790f298518b9d99 P 2976192c711f45768457228b28546c86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:50:31.826282 19214 mvcc.cc:205] Tried to move back new op lower bound from 7316130331908767744 to 7316130331810578432. Current Snapshot: MvccSnapshot[applied={T|T < 7316130331908767744}]
I20260808 04:50:31.829102 19218 mvcc.cc:205] Tried to move back new op lower bound from 7316130331908767744 to 7316130331810578432. Current Snapshot: MvccSnapshot[applied={T|T < 7316130331908767744}]
I20260808 04:50:31.834438 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.830525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.15.30.67:37575)
0808 04:50:31.830660 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.830664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.830679 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:31.830752 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.830755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.830810 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.830915 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:31.831146 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.831949 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.832938 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.833283 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.833394 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.833609 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:31.834079 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:31.834193 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.834258 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.834312 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:50:31.845095 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.841777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:36275 (local address 127.15.30.67:57141)
0808 04:50:31.841925 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.841947 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.842011 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.842248 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.842250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.842255 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:31.842336 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.842340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.843299 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.843302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.843880 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.843888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.844462 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.844465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.844531 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.844572 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:31.844893 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:31.844898 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.844946 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.845002 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:50:31.845129 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.841853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:57141 (local address 127.15.30.65:36275)
0808 04:50:31.841965 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.841969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.841983 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:50:31.842102 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.842106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.842154 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.842235 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:31.842437 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.843180 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.843987 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.844329 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.844466 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.844730 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:31.844808 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:31.844883 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.844958 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.845007 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:50:31.852347 18827 raft_consensus.cc:1261] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Refusing update from remote peer 8362b15555084bff89c57785ead867da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:31.852586 18588 raft_consensus.cc:1261] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Refusing update from remote peer 8362b15555084bff89c57785ead867da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:31.852859 19177 consensus_queue.cc:1035] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:31.853127 19171 consensus_queue.cc:1035] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:31.876483 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.872508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:40267 (local address 127.15.30.67:37575)
0808 04:50:31.872657 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:31.872662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:31.872912 (+ 250us) server_negotiation.cc:407] Connection header received
0808 04:50:31.873048 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:31.873050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:31.873089 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:31.873160 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:31.873496 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.874165 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.875249 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.875583 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.875701 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.875999 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:31.876073 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:31.876151 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:31.876252 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:50:31.876318 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:50:31.876564 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:31.872460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37575 (local address 127.15.30.65:40267)
0808 04:50:31.872833 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:31.872848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:31.872969 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:31.873193 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:31.873195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:31.873203 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:31.873318 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.873324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.874278 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.874283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.875121 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:31.875132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:31.875702 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:31.875705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:31.875812 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:31.875855 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:31.876171 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:31.876178 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:50:31.876242 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 04:50:31.876320 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":139,"threads_started":1}
I20260808 04:50:31.899883 18376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:31.900077 18376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:31.902758 18376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Sending DeleteTablet for 3 replicas of tablet eaa4a0b970334e889c0e8c816f1de8b2
I20260808 04:50:31.903470 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa4a0b970334e889c0e8c816f1de8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:31 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:44262
I20260808 04:50:31.903609 18792 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa4a0b970334e889c0e8c816f1de8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:31 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39462
I20260808 04:50:31.904197 18577 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa4a0b970334e889c0e8c816f1de8b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:31 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42294
I20260808 04:50:31.904749 19245 tablet_replica.cc:330] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: stopping tablet replica
I20260808 04:50:31.904873 18376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:31.905282 18376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:31.905297 19245 raft_consensus.cc:2229] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:31.905601 19245 raft_consensus.cc:2258] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:31.907033 19245 ts_tablet_manager.cc:1905] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:31.908083 19242 tablet_replica.cc:330] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: stopping tablet replica
I20260808 04:50:31.908404 19242 raft_consensus.cc:2229] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:31.908612 19242 raft_consensus.cc:2258] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:31.910599 18577 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 (DEFAULT_TABLE table=test-table [id=9140eef3385d4f0bbfb4e7b96d3703f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:31.910888 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.911801 19245 ts_tablet_manager.cc:1918] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:31.912621 19017 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 (DEFAULT_TABLE table=test-table [id=9140eef3385d4f0bbfb4e7b96d3703f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:31.912815 19245 log.cc:1196] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/eaa4a0b970334e889c0e8c816f1de8b2
I20260808 04:50:31.912849 18792 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 (DEFAULT_TABLE table=test-table [id=9140eef3385d4f0bbfb4e7b96d3703f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:31.912918 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.913122 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:31.913385 19245 ts_tablet_manager.cc:1939] T eaa4a0b970334e889c0e8c816f1de8b2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting consensus metadata
I20260808 04:50:31.915477 19163 tablet_bootstrap.cc:492] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: Bootstrap starting.
I20260808 04:50:31.915612 19164 tablet_bootstrap.cc:492] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: Bootstrap starting.
I20260808 04:50:31.916328 19163 tablet_bootstrap.cc:654] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.916620 19164 tablet_bootstrap.cc:654] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.912598 19241 tablet_replica.cc:330] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: stopping tablet replica
I20260808 04:50:31.917470 19241 raft_consensus.cc:2229] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:31.917526 18355 catalog_manager.cc:4841] TS 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275): tablet eaa4a0b970334e889c0e8c816f1de8b2 (table test-table [id=526211ce4b54460cb8ae4f89f43ce1c6]) successfully deleted
I20260808 04:50:31.917685 19163 tablet_bootstrap.cc:492] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: No bootstrap required, opened a new log
I20260808 04:50:31.917757 19163 ts_tablet_manager.cc:1397] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 04:50:31.917807 19241 raft_consensus.cc:2258] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:31.918205 19163 raft_consensus.cc:348] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.918308 19163 raft_consensus.cc:374] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.918339 19163 raft_consensus.cc:729] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8362b15555084bff89c57785ead867da, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.918447 19163 consensus_queue.cc:260] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.918687 19161 tablet_bootstrap.cc:492] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Bootstrap starting.
I20260808 04:50:31.918704 19163 ts_tablet_manager.cc:1428] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:50:31.919667 19161 tablet_bootstrap.cc:654] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:31.920542 19241 ts_tablet_manager.cc:1905] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:31.921517 19242 ts_tablet_manager.cc:1905] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:31.924346 19242 ts_tablet_manager.cc:1918] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:31.924463 19242 log.cc:1196] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/eaa4a0b970334e889c0e8c816f1de8b2
I20260808 04:50:31.924790 19242 ts_tablet_manager.cc:1939] T eaa4a0b970334e889c0e8c816f1de8b2 P 2976192c711f45768457228b28546c86: Deleting consensus metadata
I20260808 04:50:31.925976 18353 catalog_manager.cc:4841] TS 2976192c711f45768457228b28546c86 (127.15.30.66:39933): tablet eaa4a0b970334e889c0e8c816f1de8b2 (table test-table [id=526211ce4b54460cb8ae4f89f43ce1c6]) successfully deleted
I20260808 04:50:31.926465 19164 tablet_bootstrap.cc:492] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: No bootstrap required, opened a new log
I20260808 04:50:31.926556 19164 ts_tablet_manager.cc:1397] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260808 04:50:31.927003 19164 raft_consensus.cc:348] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.927125 19164 raft_consensus.cc:374] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.927177 19164 raft_consensus.cc:729] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2976192c711f45768457228b28546c86, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.927338 19164 consensus_queue.cc:260] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.927662 19164 ts_tablet_manager.cc:1428] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:31.929153 19161 tablet_bootstrap.cc:492] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: No bootstrap required, opened a new log
I20260808 04:50:31.929241 19161 ts_tablet_manager.cc:1397] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260808 04:50:31.929665 19161 raft_consensus.cc:348] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.929813 19161 raft_consensus.cc:374] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:31.929849 19161 raft_consensus.cc:729] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2739ca8d72d44e4db69a4ee62fc45e9e, State: Initialized, Role: FOLLOWER
I20260808 04:50:31.929982 19161 consensus_queue.cc:260] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:31.930251 19161 ts_tablet_manager.cc:1428] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:31.931305 19241 ts_tablet_manager.cc:1918] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:31.931447 19241 log.cc:1196] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/eaa4a0b970334e889c0e8c816f1de8b2
I20260808 04:50:31.931828 19241 ts_tablet_manager.cc:1939] T eaa4a0b970334e889c0e8c816f1de8b2 P 8362b15555084bff89c57785ead867da: Deleting consensus metadata
I20260808 04:50:31.933009 18355 catalog_manager.cc:4841] TS 8362b15555084bff89c57785ead867da (127.15.30.67:37575): tablet eaa4a0b970334e889c0e8c816f1de8b2 (table test-table [id=526211ce4b54460cb8ae4f89f43ce1c6]) successfully deleted
I20260808 04:50:32.040838 19173 raft_consensus.cc:482] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:32.040997 19173 raft_consensus.cc:504] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:32.042316 19173 leader_election.cc:288] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275), 8362b15555084bff89c57785ead867da (127.15.30.67:37575)
I20260808 04:50:32.047055 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.043034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:58643 (local address 127.15.30.67:37575)
0808 04:50:32.043150 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:32.043155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:32.043389 (+ 234us) server_negotiation.cc:407] Connection header received
0808 04:50:32.043493 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:32.043497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:32.043553 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:32.043662 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:32.043964 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.044749 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.046048 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.046438 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.046569 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.046630 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:32.046827 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:50:32.046918 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:50:32.047102 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.042913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37575 (local address 127.15.30.66:58643)
0808 04:50:32.043272 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:32.043286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:32.043404 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:32.043715 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:32.043718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:32.043738 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:32.043835 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.043841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.044878 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.044882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.045934 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.045941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.046563 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.046566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.046682 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.046705 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:50:32.046794 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:50:32.046937 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":46,"thread_start_us":94,"threads_started":1}
I20260808 04:50:32.047201 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.042660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:45233 (local address 127.15.30.65:36275)
0808 04:50:32.042818 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:32.042823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:32.042847 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:50:32.043054 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:32.043057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:32.043121 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:32.043182 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:32.043668 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.044348 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.045834 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.046456 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.046602 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.046661 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:32.046974 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:50:32.047069 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:50:32.047577 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc84c688b684c2a8a51e8c5cf0dd9c2" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8362b15555084bff89c57785ead867da" is_pre_election: true
I20260808 04:50:32.047766 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.042513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:36275 (local address 127.15.30.66:45233)
0808 04:50:32.042781 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:32.042798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:32.042894 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:32.043404 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:32.043407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:32.043421 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:32.043553 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.043558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.044512 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.044517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.045680 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.045691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.046700 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.046703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.046805 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.046850 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:50:32.047532 (+ 682us) client_negotiation.cc:240] Negotiation successful
0808 04:50:32.047607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260808 04:50:32.047756 19048 raft_consensus.cc:2454] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 0.
I20260808 04:50:32.048087 18717 leader_election.cc:302] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2976192c711f45768457228b28546c86, 8362b15555084bff89c57785ead867da; no voters:
I20260808 04:50:32.048449 19173 raft_consensus.cc:2781] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:32.048521 19173 raft_consensus.cc:482] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:32.048563 19173 raft_consensus.cc:3037] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:32.048689 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc84c688b684c2a8a51e8c5cf0dd9c2" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" is_pre_election: true
I20260808 04:50:32.048815 18588 raft_consensus.cc:2454] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 0.
I20260808 04:50:32.049926 19173 raft_consensus.cc:504] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:32.050339 19173 leader_election.cc:288] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 election: Requested vote from peers 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275), 8362b15555084bff89c57785ead867da (127.15.30.67:37575)
I20260808 04:50:32.050704 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc84c688b684c2a8a51e8c5cf0dd9c2" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e"
I20260808 04:50:32.050895 18588 raft_consensus.cc:3037] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:32.050956 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc84c688b684c2a8a51e8c5cf0dd9c2" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8362b15555084bff89c57785ead867da"
I20260808 04:50:32.051081 19048 raft_consensus.cc:3037] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:32.052250 18588 raft_consensus.cc:2454] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 1.
I20260808 04:50:32.052531 19048 raft_consensus.cc:2454] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 1.
I20260808 04:50:32.052593 18717 leader_election.cc:302] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2739ca8d72d44e4db69a4ee62fc45e9e, 2976192c711f45768457228b28546c86; no voters:
I20260808 04:50:32.052896 19173 raft_consensus.cc:2781] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:32.053113 19173 raft_consensus.cc:686] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 1 LEADER]: Becoming Leader. State: Replica: 2976192c711f45768457228b28546c86, State: Running, Role: LEADER
I20260808 04:50:32.053653 19173 consensus_queue.cc:237] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:32.056538 18376 catalog_manager.cc:5495] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2976192c711f45768457228b28546c86 (127.15.30.66). New cstate: current_term: 1 leader_uuid: "2976192c711f45768457228b28546c86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } health_report { overall_health: UNKNOWN } } }
I20260808 04:50:32.079066 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.074416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.15.30.66:39933)
0808 04:50:32.074559 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:32.074563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:32.074577 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:50:32.074646 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:32.074649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:32.074705 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:32.074801 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:32.075153 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.075929 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.077204 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.077533 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.077628 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.077958 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:32.078474 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:32.078576 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:32.078873 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 04:50:32.078925 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:50:32.090801 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.087263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:55029 (local address 127.15.30.65:36275)
0808 04:50:32.087367 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:32.087371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:32.087672 (+ 301us) server_negotiation.cc:407] Connection header received
0808 04:50:32.087820 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:32.087823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:32.087873 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:32.087951 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:32.088176 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.088865 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.089707 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.089991 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.090078 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.090389 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:32.090477 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:32.090528 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:32.090663 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 04:50:32.090705 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:50:32.091090 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.087165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:36275 (local address 127.15.30.66:55029)
0808 04:50:32.087553 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:32.087586 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:32.087716 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:32.087972 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:32.087974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:32.087980 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:32.088074 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.088078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.088972 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.088975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.089578 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.089586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.090148 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.090152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.090223 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.090265 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:32.090566 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:32.090574 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:50:32.090635 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 04:50:32.090698 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":157,"threads_started":1}
I20260808 04:50:32.097378 18588 raft_consensus.cc:1261] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Refusing update from remote peer 2976192c711f45768457228b28546c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:32.097456 19048 raft_consensus.cc:1261] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Refusing update from remote peer 2976192c711f45768457228b28546c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:32.097872 19253 consensus_queue.cc:1035] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:32.098074 19173 consensus_queue.cc:1035] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:32.116494 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.113304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:39933 (local address 127.15.30.65:35157)
0808 04:50:32.113427 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:32.113443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:32.113524 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:32.113744 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:32.113746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:32.113751 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:50:32.113833 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.113836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.114810 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.114812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.115382 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:32.115390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.115817 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.115819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:32.115878 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.115905 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:50:32.116196 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:50:32.116205 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:50:32.116271 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:50:32.116356 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 04:50:32.116484 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:32.113452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35157 (local address 127.15.30.66:39933)
0808 04:50:32.113570 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:32.113576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:32.113591 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:32.113636 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:32.113643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:32.113681 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:32.113740 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:32.113932 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.114717 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.115481 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:32.115701 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:32.115821 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:32.116018 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:32.116098 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:32.116183 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:32.116280 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 04:50:32.116329 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 04:50:32.117216 19218 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:50:33.677605 18376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:33.677767 18376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:33.680157 18376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Sending DeleteTablet for 3 replicas of tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2
I20260808 04:50:33.680866 18792 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:33 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39462
I20260808 04:50:33.680975 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:33 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:44262
I20260808 04:50:33.681340 19299 tablet_replica.cc:330] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: stopping tablet replica
I20260808 04:50:33.681450 19301 tablet_replica.cc:330] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: stopping tablet replica
I20260808 04:50:33.681578 19299 raft_consensus.cc:2229] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:33.681604 19301 raft_consensus.cc:2229] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:33.681715 19301 raft_consensus.cc:2258] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:33.681800 18577 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:33 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42294
I20260808 04:50:33.682015 19299 raft_consensus.cc:2258] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:33.682145 19304 tablet_replica.cc:330] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: stopping tablet replica
I20260808 04:50:33.682274 19304 raft_consensus.cc:2229] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:33.682374 19304 raft_consensus.cc:2258] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:33.682467 19301 ts_tablet_manager.cc:1905] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:33.682813 19299 ts_tablet_manager.cc:1905] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:33.685274 19301 ts_tablet_manager.cc:1918] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:50:33.685369 19301 log.cc:1196] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/1dc84c688b684c2a8a51e8c5cf0dd9c2
I20260808 04:50:33.685671 19301 ts_tablet_manager.cc:1939] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 8362b15555084bff89c57785ead867da: Deleting consensus metadata
I20260808 04:50:33.686509 18355 catalog_manager.cc:4841] TS 8362b15555084bff89c57785ead867da (127.15.30.67:37575): tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 (table test-table [id=9140eef3385d4f0bbfb4e7b96d3703f8]) successfully deleted
I20260808 04:50:33.686586 19299 ts_tablet_manager.cc:1918] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:50:33.688513 19299 log.cc:1196] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/1dc84c688b684c2a8a51e8c5cf0dd9c2
I20260808 04:50:33.688589 19304 ts_tablet_manager.cc:1905] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:33.688865 19299 ts_tablet_manager.cc:1939] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2976192c711f45768457228b28546c86: Deleting consensus metadata
I20260808 04:50:33.689509 18353 catalog_manager.cc:4841] TS 2976192c711f45768457228b28546c86 (127.15.30.66:39933): tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 (table test-table [id=9140eef3385d4f0bbfb4e7b96d3703f8]) successfully deleted
I20260808 04:50:33.691138 19304 ts_tablet_manager.cc:1918] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:50:33.691246 19304 log.cc:1196] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/1dc84c688b684c2a8a51e8c5cf0dd9c2
I20260808 04:50:33.691511 19304 ts_tablet_manager.cc:1939] T 1dc84c688b684c2a8a51e8c5cf0dd9c2 P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting consensus metadata
I20260808 04:50:33.692198 18355 catalog_manager.cc:4841] TS 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275): tablet 1dc84c688b684c2a8a51e8c5cf0dd9c2 (table test-table [id=9140eef3385d4f0bbfb4e7b96d3703f8]) successfully deleted
Aug 08 04:50:33 dist-test-slave-zpvx krb5kdc[18309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:50:33 dist-test-slave-zpvx krb5kdc[18309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:33.711164 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:33.702803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.15.30.126:44067)
0808 04:50:33.703083 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:33.703089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:33.703102 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:50:33.703210 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:33.703212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:33.703252 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:33.703327 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:33.703977 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:33.704622 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:33.705414 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:33.705678 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:33.705759 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:33.708257 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:33.708260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:33.708261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:33.708284 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:33.709668 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:33.710044 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:33.710046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:33.710048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:33.710122 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:33.710352 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:33.710354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:33.710355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:33.710529 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:33.710631 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:33.710837 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 04:50:33.710936 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260808 04:50:33.713711 18376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:33.714154 18376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:33.718945 19017 tablet_service.cc:1467] Processing CreateTablet for tablet e7bc6e41fede4ee5b2d63d58bb221c9d (DEFAULT_TABLE table=test-table [id=1b326ba090774412b101cd4471d440cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:33.718945 18577 tablet_service.cc:1467] Processing CreateTablet for tablet e7bc6e41fede4ee5b2d63d58bb221c9d (DEFAULT_TABLE table=test-table [id=1b326ba090774412b101cd4471d440cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:33.718971 18792 tablet_service.cc:1467] Processing CreateTablet for tablet e7bc6e41fede4ee5b2d63d58bb221c9d (DEFAULT_TABLE table=test-table [id=1b326ba090774412b101cd4471d440cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:33.719237 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bc6e41fede4ee5b2d63d58bb221c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:33.719249 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bc6e41fede4ee5b2d63d58bb221c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:33.719272 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bc6e41fede4ee5b2d63d58bb221c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:33.722040 19318 tablet_bootstrap.cc:492] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: Bootstrap starting.
I20260808 04:50:33.722097 19319 tablet_bootstrap.cc:492] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: Bootstrap starting.
I20260808 04:50:33.722292 19324 tablet_bootstrap.cc:492] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: Bootstrap starting.
I20260808 04:50:33.722920 19318 tablet_bootstrap.cc:654] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:33.722920 19319 tablet_bootstrap.cc:654] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:33.723038 19324 tablet_bootstrap.cc:654] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:33.724167 19319 tablet_bootstrap.cc:492] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: No bootstrap required, opened a new log
I20260808 04:50:33.724292 19319 ts_tablet_manager.cc:1397] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:33.724553 19318 tablet_bootstrap.cc:492] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: No bootstrap required, opened a new log
I20260808 04:50:33.724637 19318 ts_tablet_manager.cc:1397] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:33.724763 19324 tablet_bootstrap.cc:492] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: No bootstrap required, opened a new log
I20260808 04:50:33.724748 19319 raft_consensus.cc:348] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.724857 19324 ts_tablet_manager.cc:1397] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:33.724865 19319 raft_consensus.cc:374] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:33.724900 19319 raft_consensus.cc:729] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2739ca8d72d44e4db69a4ee62fc45e9e, State: Initialized, Role: FOLLOWER
I20260808 04:50:33.725055 19319 consensus_queue.cc:260] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.725049 19318 raft_consensus.cc:348] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.725164 19318 raft_consensus.cc:374] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:33.725198 19318 raft_consensus.cc:729] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8362b15555084bff89c57785ead867da, State: Initialized, Role: FOLLOWER
I20260808 04:50:33.725296 19324 raft_consensus.cc:348] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.725329 19318 consensus_queue.cc:260] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.725421 19324 raft_consensus.cc:374] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:33.725463 19324 raft_consensus.cc:729] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2976192c711f45768457228b28546c86, State: Initialized, Role: FOLLOWER
I20260808 04:50:33.725601 19324 consensus_queue.cc:260] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.725809 19318 ts_tablet_manager.cc:1428] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:33.726253 19324 ts_tablet_manager.cc:1428] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:33.727109 19319 ts_tablet_manager.cc:1428] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260808 04:50:33.820549 19332 raft_consensus.cc:482] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:33.820762 19332 raft_consensus.cc:504] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.821257 19332 leader_election.cc:288] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275), 2976192c711f45768457228b28546c86 (127.15.30.66:39933)
I20260808 04:50:33.821698 18827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bc6e41fede4ee5b2d63d58bb221c9d" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2976192c711f45768457228b28546c86" is_pre_election: true
I20260808 04:50:33.821699 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bc6e41fede4ee5b2d63d58bb221c9d" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" is_pre_election: true
I20260808 04:50:33.821867 18827 raft_consensus.cc:2454] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 0.
I20260808 04:50:33.821885 18588 raft_consensus.cc:2454] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 0.
I20260808 04:50:33.822247 18935 leader_election.cc:302] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2976192c711f45768457228b28546c86, 8362b15555084bff89c57785ead867da; no voters:
I20260808 04:50:33.822479 19332 raft_consensus.cc:2781] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:33.822552 19332 raft_consensus.cc:482] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:33.822593 19332 raft_consensus.cc:3037] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:33.823622 19332 raft_consensus.cc:504] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.823900 19332 leader_election.cc:288] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 election: Requested vote from peers 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275), 2976192c711f45768457228b28546c86 (127.15.30.66:39933)
I20260808 04:50:33.824272 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bc6e41fede4ee5b2d63d58bb221c9d" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e"
I20260808 04:50:33.824276 18827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bc6e41fede4ee5b2d63d58bb221c9d" candidate_uuid: "8362b15555084bff89c57785ead867da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2976192c711f45768457228b28546c86"
I20260808 04:50:33.824419 18588 raft_consensus.cc:3037] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:33.824468 18827 raft_consensus.cc:3037] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:33.825480 18588 raft_consensus.cc:2454] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 1.
I20260808 04:50:33.825594 18827 raft_consensus.cc:2454] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8362b15555084bff89c57785ead867da in term 1.
I20260808 04:50:33.825826 18939 leader_election.cc:302] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2739ca8d72d44e4db69a4ee62fc45e9e, 8362b15555084bff89c57785ead867da; no voters:
I20260808 04:50:33.825989 19332 raft_consensus.cc:2781] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:33.826237 19332 raft_consensus.cc:686] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 1 LEADER]: Becoming Leader. State: Replica: 8362b15555084bff89c57785ead867da, State: Running, Role: LEADER
I20260808 04:50:33.826426 19332 consensus_queue.cc:237] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.828094 18373 catalog_manager.cc:5495] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da reported cstate change: term changed from 0 to 1, leader changed from <none> to 8362b15555084bff89c57785ead867da (127.15.30.67). New cstate: current_term: 1 leader_uuid: "8362b15555084bff89c57785ead867da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } health_report { overall_health: HEALTHY } } }
I20260808 04:50:33.851099 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:33.847657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.15.30.67:37575)
0808 04:50:33.848091 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:33.848103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:33.848117 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:50:33.848162 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:33.848164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:33.848213 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:33.848306 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:33.848581 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:33.849184 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:33.850042 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:33.850266 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:33.850372 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:33.850625 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:33.850691 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:33.850770 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:33.850896 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:50:33.850949 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":169,"threads_started":1}
I20260808 04:50:33.857165 18827 raft_consensus.cc:1261] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Refusing update from remote peer 8362b15555084bff89c57785ead867da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:33.857308 18588 raft_consensus.cc:1261] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Refusing update from remote peer 8362b15555084bff89c57785ead867da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:33.857633 19334 consensus_queue.cc:1035] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:33.857820 19332 consensus_queue.cc:1035] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:33.889609 18373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:33.889755 18373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:33.891747 18373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Sending DeleteTablet for 3 replicas of tablet e7bc6e41fede4ee5b2d63d58bb221c9d
I20260808 04:50:33.892292 18577 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc6e41fede4ee5b2d63d58bb221c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:33 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42294
I20260808 04:50:33.892408 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc6e41fede4ee5b2d63d58bb221c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:33 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:44262
I20260808 04:50:33.892407 18792 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc6e41fede4ee5b2d63d58bb221c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:33 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39462
I20260808 04:50:33.892501 19304 tablet_replica.cc:330] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: stopping tablet replica
I20260808 04:50:33.892565 19301 tablet_replica.cc:330] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: stopping tablet replica
I20260808 04:50:33.892587 19299 tablet_replica.cc:330] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: stopping tablet replica
I20260808 04:50:33.892629 19304 raft_consensus.cc:2229] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:33.892671 19301 raft_consensus.cc:2229] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:33.892704 19299 raft_consensus.cc:2229] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:33.892752 19304 raft_consensus.cc:2258] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:33.892820 19299 raft_consensus.cc:2258] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:33.892855 19301 raft_consensus.cc:2258] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:33.893649 19304 ts_tablet_manager.cc:1905] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:33.893652 19299 ts_tablet_manager.cc:1905] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:33.893940 18373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:33.894457 18373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:33.895664 19304 ts_tablet_manager.cc:1918] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:33.895735 19304 log.cc:1196] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/e7bc6e41fede4ee5b2d63d58bb221c9d
I20260808 04:50:33.895972 19304 ts_tablet_manager.cc:1939] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2739ca8d72d44e4db69a4ee62fc45e9e: Deleting consensus metadata
I20260808 04:50:33.896549 19299 ts_tablet_manager.cc:1918] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:33.896654 19299 log.cc:1196] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/e7bc6e41fede4ee5b2d63d58bb221c9d
I20260808 04:50:33.896642 18355 catalog_manager.cc:4841] TS 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275): tablet e7bc6e41fede4ee5b2d63d58bb221c9d (table test-table [id=1b326ba090774412b101cd4471d440cc]) successfully deleted
I20260808 04:50:33.896935 19299 ts_tablet_manager.cc:1939] T e7bc6e41fede4ee5b2d63d58bb221c9d P 2976192c711f45768457228b28546c86: Deleting consensus metadata
I20260808 04:50:33.897591 18353 catalog_manager.cc:4841] TS 2976192c711f45768457228b28546c86 (127.15.30.66:39933): tablet e7bc6e41fede4ee5b2d63d58bb221c9d (table test-table [id=1b326ba090774412b101cd4471d440cc]) successfully deleted
I20260808 04:50:33.897840 19301 ts_tablet_manager.cc:1905] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:33.899351 18577 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5db22bff0c49109939b26387972547 (DEFAULT_TABLE table=test-table [id=496011953e744e708220c432e3820a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:33.899474 19017 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5db22bff0c49109939b26387972547 (DEFAULT_TABLE table=test-table [id=496011953e744e708220c432e3820a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:33.899601 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5db22bff0c49109939b26387972547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:33.899574 18792 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5db22bff0c49109939b26387972547 (DEFAULT_TABLE table=test-table [id=496011953e744e708220c432e3820a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:33.899712 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5db22bff0c49109939b26387972547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:33.899791 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5db22bff0c49109939b26387972547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:33.900516 19301 ts_tablet_manager.cc:1918] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:50:33.900590 19301 log.cc:1196] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/e7bc6e41fede4ee5b2d63d58bb221c9d
I20260808 04:50:33.900913 19301 ts_tablet_manager.cc:1939] T e7bc6e41fede4ee5b2d63d58bb221c9d P 8362b15555084bff89c57785ead867da: Deleting consensus metadata
I20260808 04:50:33.901947 19318 tablet_bootstrap.cc:492] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: Bootstrap starting.
I20260808 04:50:33.902043 19319 tablet_bootstrap.cc:492] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e: Bootstrap starting.
I20260808 04:50:33.902232 19324 tablet_bootstrap.cc:492] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: Bootstrap starting.
I20260808 04:50:33.902794 19318 tablet_bootstrap.cc:654] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:33.902832 19319 tablet_bootstrap.cc:654] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:33.902966 19324 tablet_bootstrap.cc:654] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:33.903903 18355 catalog_manager.cc:4841] TS 8362b15555084bff89c57785ead867da (127.15.30.67:37575): tablet e7bc6e41fede4ee5b2d63d58bb221c9d (table test-table [id=1b326ba090774412b101cd4471d440cc]) successfully deleted
I20260808 04:50:33.904156 19319 tablet_bootstrap.cc:492] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e: No bootstrap required, opened a new log
I20260808 04:50:33.904240 19319 ts_tablet_manager.cc:1397] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:33.904680 19319 raft_consensus.cc:348] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.904778 19319 raft_consensus.cc:374] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:33.904799 19319 raft_consensus.cc:729] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2739ca8d72d44e4db69a4ee62fc45e9e, State: Initialized, Role: FOLLOWER
I20260808 04:50:33.904821 19318 tablet_bootstrap.cc:492] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: No bootstrap required, opened a new log
I20260808 04:50:33.904908 19318 ts_tablet_manager.cc:1397] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 04:50:33.904920 19319 consensus_queue.cc:260] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.905207 19319 ts_tablet_manager.cc:1428] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:33.905299 19318 raft_consensus.cc:348] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.905400 19318 raft_consensus.cc:374] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:33.905433 19318 raft_consensus.cc:729] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8362b15555084bff89c57785ead867da, State: Initialized, Role: FOLLOWER
I20260808 04:50:33.905555 19318 consensus_queue.cc:260] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.905834 19318 ts_tablet_manager.cc:1428] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:50:33.906765 19324 tablet_bootstrap.cc:492] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: No bootstrap required, opened a new log
I20260808 04:50:33.906854 19324 ts_tablet_manager.cc:1397] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 04:50:33.907251 19324 raft_consensus.cc:348] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.907392 19324 raft_consensus.cc:374] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:33.907425 19324 raft_consensus.cc:729] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2976192c711f45768457228b28546c86, State: Initialized, Role: FOLLOWER
I20260808 04:50:33.907573 19324 consensus_queue.cc:260] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.907850 19324 ts_tablet_manager.cc:1428] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:33.935874 19288 raft_consensus.cc:482] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:33.936009 19288 raft_consensus.cc:504] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.936425 19288 leader_election.cc:288] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275), 8362b15555084bff89c57785ead867da (127.15.30.67:37575)
I20260808 04:50:33.936839 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5db22bff0c49109939b26387972547" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" is_pre_election: true
I20260808 04:50:33.936966 18588 raft_consensus.cc:2454] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 0.
I20260808 04:50:33.936946 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5db22bff0c49109939b26387972547" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8362b15555084bff89c57785ead867da" is_pre_election: true
I20260808 04:50:33.937157 19047 raft_consensus.cc:2454] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 0.
I20260808 04:50:33.937314 18717 leader_election.cc:302] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2739ca8d72d44e4db69a4ee62fc45e9e, 2976192c711f45768457228b28546c86; no voters:
I20260808 04:50:33.937542 19288 raft_consensus.cc:2781] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:33.937618 19288 raft_consensus.cc:482] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:33.937672 19288 raft_consensus.cc:3037] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:33.938665 19288 raft_consensus.cc:504] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.938990 19288 leader_election.cc:288] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 election: Requested vote from peers 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275), 8362b15555084bff89c57785ead867da (127.15.30.67:37575)
I20260808 04:50:33.939319 18588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5db22bff0c49109939b26387972547" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e"
I20260808 04:50:33.939375 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5db22bff0c49109939b26387972547" candidate_uuid: "2976192c711f45768457228b28546c86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8362b15555084bff89c57785ead867da"
I20260808 04:50:33.939429 18588 raft_consensus.cc:3037] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:33.939514 19047 raft_consensus.cc:3037] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:33.940333 18588 raft_consensus.cc:2454] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 1.
I20260808 04:50:33.940511 19047 raft_consensus.cc:2454] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2976192c711f45768457228b28546c86 in term 1.
I20260808 04:50:33.940680 18717 leader_election.cc:302] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2739ca8d72d44e4db69a4ee62fc45e9e, 2976192c711f45768457228b28546c86; no voters:
I20260808 04:50:33.940896 19288 raft_consensus.cc:2781] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:33.941008 19288 raft_consensus.cc:686] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 1 LEADER]: Becoming Leader. State: Replica: 2976192c711f45768457228b28546c86, State: Running, Role: LEADER
I20260808 04:50:33.941141 19288 consensus_queue.cc:237] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } }
I20260808 04:50:33.942691 18373 catalog_manager.cc:5495] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2976192c711f45768457228b28546c86 (127.15.30.66). New cstate: current_term: 1 leader_uuid: "2976192c711f45768457228b28546c86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2976192c711f45768457228b28546c86" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 } health_report { overall_health: UNKNOWN } } }
I20260808 04:50:33.966286 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:33.963188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.15.30.66:39933)
0808 04:50:33.963473 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:33.963479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:33.963492 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:50:33.963529 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:33.963531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:33.963569 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:33.963650 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:33.963866 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:33.964436 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:33.965363 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:33.965611 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:33.965695 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:33.965906 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:33.965977 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:33.966029 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:33.966132 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 04:50:33.966176 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":141,"threads_started":1}
I20260808 04:50:33.971820 18588 raft_consensus.cc:1261] T 1c5db22bff0c49109939b26387972547 P 2739ca8d72d44e4db69a4ee62fc45e9e [term 1 FOLLOWER]: Refusing update from remote peer 2976192c711f45768457228b28546c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:33.971851 19047 raft_consensus.cc:1261] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Refusing update from remote peer 2976192c711f45768457228b28546c86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:33.972221 19262 consensus_queue.cc:1035] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8362b15555084bff89c57785ead867da" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:33.972563 19262 consensus_queue.cc:1035] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2739ca8d72d44e4db69a4ee62fc45e9e" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 36275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:33.977283 19268 mvcc.cc:205] Tried to move back new op lower bound from 7316130340745367552 to 7316130340624142336. Current Snapshot: MvccSnapshot[applied={T|T < 7316130340745367552}]
W20260808 04:50:33.985370 19272 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:50:35.535126 18372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:35.535305 18372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:35.537760 18372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73e8823ca278490bb5691d8aefc2a467: Sending DeleteTablet for 3 replicas of tablet 1c5db22bff0c49109939b26387972547
I20260808 04:50:35.538255 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5db22bff0c49109939b26387972547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42294
I20260808 04:50:35.538317 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5db22bff0c49109939b26387972547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:44262
I20260808 04:50:35.538398 18792 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5db22bff0c49109939b26387972547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39462
I20260808 04:50:35.538640 19400 tablet_replica.cc:330] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: stopping tablet replica
I20260808 04:50:35.538756 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18485
I20260808 04:50:35.538771 19400 raft_consensus.cc:2229] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:35.541705 19400 raft_consensus.cc:2258] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:35.541926 19401 tablet_replica.cc:330] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: stopping tablet replica
I20260808 04:50:35.542052 19401 raft_consensus.cc:2229] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:35.542218 19401 raft_consensus.cc:2258] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:35.543859 19401 ts_tablet_manager.cc:1905] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:35.545660 19400 ts_tablet_manager.cc:1905] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:35.546782 19401 ts_tablet_manager.cc:1918] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:50:35.546849 19401 log.cc:1196] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/1c5db22bff0c49109939b26387972547
I20260808 04:50:35.547164 19401 ts_tablet_manager.cc:1939] T 1c5db22bff0c49109939b26387972547 P 2976192c711f45768457228b28546c86: Deleting consensus metadata
I20260808 04:50:35.547820 18353 catalog_manager.cc:4841] TS 2976192c711f45768457228b28546c86 (127.15.30.66:39933): tablet 1c5db22bff0c49109939b26387972547 (table test-table [id=496011953e744e708220c432e3820a64]) successfully deleted
I20260808 04:50:35.548192 19400 ts_tablet_manager.cc:1918] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:50:35.548245 19400 log.cc:1196] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/1c5db22bff0c49109939b26387972547
I20260808 04:50:35.548524 19400 ts_tablet_manager.cc:1939] T 1c5db22bff0c49109939b26387972547 P 8362b15555084bff89c57785ead867da: Deleting consensus metadata
I20260808 04:50:35.549163 18355 catalog_manager.cc:4841] TS 8362b15555084bff89c57785ead867da (127.15.30.67:37575): tablet 1c5db22bff0c49109939b26387972547 (table test-table [id=496011953e744e708220c432e3820a64]) successfully deleted
W20260808 04:50:35.558053 18355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:36275 (error 108)
I20260808 04:50:35.558302 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18695
I20260808 04:50:35.563421 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:35.562436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:36275 (local address 127.0.0.1:38834)
0808 04:50:35.563152 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:35.563271 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:36275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":443,"threads_started":1}
W20260808 04:50:35.563690 18355 catalog_manager.cc:4584] TS 2739ca8d72d44e4db69a4ee62fc45e9e (127.15.30.65:36275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c5db22bff0c49109939b26387972547: Network error: Client connection negotiation failed: client connection to 127.15.30.65:36275: connect: Connection refused (error 111)
I20260808 04:50:35.570271 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18912
I20260808 04:50:35.587435 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18329
2026-08-08T04:50:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6446 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:35 dist-test-slave-zpvx krb5kdc[19408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:35 dist-test-slave-zpvx krb5kdc[19408](info): set up 2 sockets
Aug 08 04:50:35 dist-test-slave-zpvx krb5kdc[19408](info): commencing operation
krb5kdc: starting...
W20260808 04:50:36.863015 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:36 dist-test-slave-zpvx krb5kdc[19408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:36.987061 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:46591
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:46591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:37.101326 19431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:37.101612 19431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:37.101699 19431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:37.105654 19431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:37.105738 19431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:37.105777 19431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:37.105837 19431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:37.105875 19431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:37.111020 19431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:46591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:37.112061 19431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:37.113129 19431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:37.121539 19440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:37.121516 19443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:37.121654 19437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:37.121850 19431 server_base.cc:1034] running on GCE node
I20260808 04:50:37.122498 19431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:37.123517 19431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:37.124706 19431 hybrid_clock.cc:648] HybridClock initialized: now 1786164637124681 us; error 38 us; skew 500 ppm
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:37.133028 19431 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:37.133898 19431 webserver.cc:460] Webserver started at http://127.15.30.126:43485/ using document root <none> and password file <none>
I20260808 04:50:37.134462 19431 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:37.134558 19431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:37.134795 19431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:37.136917 19431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "1c92316bd0fc48399b11418d875b1af2"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "6ad447bb79157c7a0ea5e935fbe118f2"
server_key_iv: "7a7196eba9ae7d094830f7ff89e5846b"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.137506 19431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "1c92316bd0fc48399b11418d875b1af2"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "6ad447bb79157c7a0ea5e935fbe118f2"
server_key_iv: "7a7196eba9ae7d094830f7ff89e5846b"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.141181 19431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:50:37.143587 19451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.144799 19431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:37.144939 19431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "1c92316bd0fc48399b11418d875b1af2"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "6ad447bb79157c7a0ea5e935fbe118f2"
server_key_iv: "7a7196eba9ae7d094830f7ff89e5846b"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.145049 19431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:37.167441 19431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:37.170674 19431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:37.170886 19431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:37.178001 19431 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:46591
I20260808 04:50:37.178048 19544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:46591 every 8 connection(s)
I20260808 04:50:37.178926 19431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:37.181627 19545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:37.182869 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 19431
I20260808 04:50:37.182962 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:37.183247 15481 external_mini_cluster.cc:1428] Setting key 40fe6d91533f5650248fc31fd1cb32d8
I20260808 04:50:37.188356 19545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2: Bootstrap starting.
I20260808 04:50:37.191010 19545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:37.191804 19545 log.cc:824] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:37.194048 19545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2: No bootstrap required, opened a new log
I20260808 04:50:37.198936 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:37.184667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.15.30.126:46591)
0808 04:50:37.185092 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:37.185111 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:37.185164 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:50:37.185972 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:37.185995 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:37.186426 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:37.186799 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:37.187532 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.188964 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.189879 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.190544 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.190703 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:37.193546 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:37.193553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:37.193570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:37.193603 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:37.195742 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.196202 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.196208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.196214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.196397 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.196687 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.196691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.196694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.197215 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:37.197401 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:37.197681 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 04:50:37.197974 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260808 04:50:37.198904 19545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c92316bd0fc48399b11418d875b1af2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46591 } }
I20260808 04:50:37.199198 19545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:37.199259 19545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c92316bd0fc48399b11418d875b1af2, State: Initialized, Role: FOLLOWER
I20260808 04:50:37.199726 19545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c92316bd0fc48399b11418d875b1af2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46591 } }
I20260808 04:50:37.199882 19545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:37.199995 19545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:37.200109 19545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:37.201122 19545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c92316bd0fc48399b11418d875b1af2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46591 } }
I20260808 04:50:37.201469 19545 leader_election.cc:302] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c92316bd0fc48399b11418d875b1af2; no voters:
I20260808 04:50:37.201771 19545 leader_election.cc:288] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:37.201897 19551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:37.202131 19551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c92316bd0fc48399b11418d875b1af2, State: Running, Role: LEADER
I20260808 04:50:37.202531 19551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c92316bd0fc48399b11418d875b1af2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46591 } }
I20260808 04:50:37.202998 19545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:37.204191 19552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c92316bd0fc48399b11418d875b1af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c92316bd0fc48399b11418d875b1af2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46591 } } }
I20260808 04:50:37.204327 19552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:37.204717 19553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c92316bd0fc48399b11418d875b1af2. Latest consensus state: current_term: 1 leader_uuid: "1c92316bd0fc48399b11418d875b1af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c92316bd0fc48399b11418d875b1af2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46591 } } }
I20260808 04:50:37.204804 19566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:37.204912 19553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:37.207618 19566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:37.212920 19566 catalog_manager.cc:1347] Generated new cluster ID: 950854540bf445a5bbdaa438a96674c5
I20260808 04:50:37.212984 19566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:37.233623 19566 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:37.234511 19566 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:37.240260 19566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c92316bd0fc48399b11418d875b1af2: Generated new TSK 0
I20260808 04:50:37.240887 19566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:37.270812 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46591
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:37.381505 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:37.381726 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:37.381772 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:37.385980 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:37.386041 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:37.386157 19587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:37.391496 19587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:37.392524 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:37.393500 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:37.403095 19602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:37.403177 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:37.403115 19587 server_base.cc:1034] running on GCE node
W20260808 04:50:37.403220 19599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:37.403682 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:37.404368 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:37.405573 19587 hybrid_clock.cc:648] HybridClock initialized: now 1786164637405559 us; error 28 us; skew 500 ppm
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:37.408237 19587 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:37.409104 19587 webserver.cc:460] Webserver started at http://127.15.30.65:37649/ using document root <none> and password file <none>
I20260808 04:50:37.409648 19587 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:37.409737 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:37.409958 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:37.411970 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "c5f4a390bb5846f08c1e332d851b0a14"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "5c1d557a976f0f30e731854dd37a3be1"
server_key_iv: "946c19fd2d3d807a4952be7432177a2a"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.412628 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "c5f4a390bb5846f08c1e332d851b0a14"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "5c1d557a976f0f30e731854dd37a3be1"
server_key_iv: "946c19fd2d3d807a4952be7432177a2a"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.416246 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:50:37.418644 19610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.419808 19587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:37.419940 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "c5f4a390bb5846f08c1e332d851b0a14"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "5c1d557a976f0f30e731854dd37a3be1"
server_key_iv: "946c19fd2d3d807a4952be7432177a2a"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.420046 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:37.437029 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:37.439720 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:37.439909 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:37.440512 19587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:37.441296 19587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:37.441368 19587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.441438 19587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:37.441480 19587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.450982 19587 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:33573
I20260808 04:50:37.451017 19796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:33573 every 8 connection(s)
I20260808 04:50:37.451848 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:37.456904 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 19587
I20260808 04:50:37.457033 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:37.457366 15481 external_mini_cluster.cc:1428] Setting key 76377f50bd45251acd1baf67f95011cb
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:37.466429 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:37.453688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:55821 (local address 127.15.30.126:46591)
0808 04:50:37.453844 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:37.453848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:37.454638 (+ 790us) server_negotiation.cc:407] Connection header received
0808 04:50:37.455842 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:37.455845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:37.455893 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:37.455971 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:37.457310 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.458007 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.459300 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.459548 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.459673 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:37.462534 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:37.462538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:37.462540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:37.462565 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:37.464004 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.464707 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.464710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.464711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.464792 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.465228 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.465230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.465231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.465406 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:37.465481 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:37.466180 (+ 699us) server_negotiation.cc:299] Negotiation successful
0808 04:50:37.466304 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:50:37.467656 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:37.454034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:46591 (local address 127.15.30.65:55821)
0808 04:50:37.454471 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:37.454512 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:37.455646 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:37.456180 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:37.456187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:37.456574 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:37.457146 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:37.457165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.458173 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.458176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:37.459167 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:37.459177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.459715 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.459719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:37.459897 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:37.460555 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:37.460571 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:37.462354 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:37.464156 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:37.464166 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:37.464185 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:37.464597 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:37.464896 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:37.464901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:37.464905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:37.465110 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:37.465517 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:37.465526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:37.465890 (+ 364us) client_negotiation.cc:769] Sending connection context
0808 04:50:37.466127 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 04:50:37.466510 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:50:37.469062 19797 heartbeater.cc:347] Connected to a master server at 127.15.30.126:46591
I20260808 04:50:37.469292 19797 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:37.469795 19797 heartbeater.cc:511] Master 127.15.30.126:46591 requested a full tablet report, sending...
I20260808 04:50:37.471426 19482 ts_manager.cc:194] Registered new tserver with Master: c5f4a390bb5846f08c1e332d851b0a14 (127.15.30.65:33573)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:50:37.473109 19482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:55821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:37.488165 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46591
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:37.598330 19804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:37.598599 19804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:37.598680 19804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:37.602697 19804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:37.602768 19804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:37.602864 19804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:37.608021 19804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:37.609005 19804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:37.609993 19804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:37.619325 19815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:37.619379 19812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:37.619364 19810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:37.619575 19804 server_base.cc:1034] running on GCE node
I20260808 04:50:37.620196 19804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:37.620886 19804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:37.622275 19804 hybrid_clock.cc:648] HybridClock initialized: now 1786164637622094 us; error 48 us; skew 500 ppm
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:37.625551 19804 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:37.626492 19804 webserver.cc:460] Webserver started at http://127.15.30.66:38547/ using document root <none> and password file <none>
I20260808 04:50:37.627079 19804 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:37.627144 19804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:37.627367 19804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:37.629647 19804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "84bbfde8fb694bc892ee98014e7c1356"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "b3aab51276abaac3dafc859820c11f3b"
server_key_iv: "d1d74f7599f0d3385575a795240ec5e9"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.630263 19804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "84bbfde8fb694bc892ee98014e7c1356"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "b3aab51276abaac3dafc859820c11f3b"
server_key_iv: "d1d74f7599f0d3385575a795240ec5e9"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.634310 19804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:50:37.636868 19827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.638092 19804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:37.638236 19804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "84bbfde8fb694bc892ee98014e7c1356"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "b3aab51276abaac3dafc859820c11f3b"
server_key_iv: "d1d74f7599f0d3385575a795240ec5e9"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.638345 19804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:37.665555 19804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:37.668324 19804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:37.668552 19804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:37.669147 19804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:37.669977 19804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:37.670059 19804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.670132 19804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:37.670187 19804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.680104 19804 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:39245
I20260808 04:50:37.680123 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:39245 every 8 connection(s)
I20260808 04:50:37.681011 19804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:37.684247 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 19804
I20260808 04:50:37.684345 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:37.684736 15481 external_mini_cluster.cc:1428] Setting key 99809f385c8180e9f0d6afb20aeb3511
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:37.699170 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:37.682784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:51869 (local address 127.15.30.126:46591)
0808 04:50:37.682920 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:37.682924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:37.683743 (+ 819us) server_negotiation.cc:407] Connection header received
0808 04:50:37.685665 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:37.685672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:37.685735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:37.685802 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:37.687644 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.688494 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.690370 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.690813 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.690944 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:37.694297 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:37.694304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:37.694310 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:37.694344 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:37.696607 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.697159 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.697163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.697165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.697253 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.697616 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.697619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.697621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.697802 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:37.698268 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:37.698868 (+ 600us) server_negotiation.cc:299] Negotiation successful
0808 04:50:37.698985 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:50:37.700047 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:37.683147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:46591 (local address 127.15.30.66:51869)
0808 04:50:37.683587 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:37.683630 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:37.685380 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:37.686265 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:37.686274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:37.686792 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:37.687491 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:37.687509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.688656 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.688662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:37.690219 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:37.690232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.691014 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.691028 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:37.691255 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:37.691871 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:37.691892 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:37.694064 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:37.696729 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:37.696734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:37.696749 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:37.697003 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:37.697356 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:37.697359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:37.697361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:37.697500 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:37.698264 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:37.698271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:37.698595 (+ 324us) client_negotiation.cc:769] Sending connection context
0808 04:50:37.698823 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 04:50:37.699129 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260808 04:50:37.701730 20002 heartbeater.cc:347] Connected to a master server at 127.15.30.126:46591
I20260808 04:50:37.702018 20002 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
I20260808 04:50:37.702651 20002 heartbeater.cc:511] Master 127.15.30.126:46591 requested a full tablet report, sending...
I20260808 04:50:37.703894 19482 ts_manager.cc:194] Registered new tserver with Master: 84bbfde8fb694bc892ee98014e7c1356 (127.15.30.66:39245)
I20260808 04:50:37.704825 19482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:51869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:37.719596 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46591
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:37.833628 20007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:37.833859 20007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:37.833899 20007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:37.837795 20007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:37.837845 20007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:37.837932 20007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:37.843003 20007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:37.843940 20007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:37.844933 20007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:37.854400 20015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:37.854439 20012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:37.854453 20007 server_base.cc:1034] running on GCE node
W20260808 04:50:37.854416 20013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:37.855062 20007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:37.855640 20007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:37.856820 20007 hybrid_clock.cc:648] HybridClock initialized: now 1786164637856808 us; error 25 us; skew 500 ppm
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:37.859614 20007 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:37.860448 20007 webserver.cc:460] Webserver started at http://127.15.30.67:40211/ using document root <none> and password file <none>
I20260808 04:50:37.860973 20007 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:37.861094 20007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:37.861313 20007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:37.863224 20007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "88de0e6b22904c2fb5bd036774ba87e3"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "463c0dc01f7045de141de739ce83c6a4"
server_key_iv: "a34f373b0275ed66eb3c524e13c85077"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.863771 20007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "88de0e6b22904c2fb5bd036774ba87e3"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "463c0dc01f7045de141de739ce83c6a4"
server_key_iv: "a34f373b0275ed66eb3c524e13c85077"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.867369 20007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:37.869884 20028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.871136 20007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:37.871290 20007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "88de0e6b22904c2fb5bd036774ba87e3"
format_stamp: "Formatted at 2026-08-08 04:50:37 on dist-test-slave-zpvx"
server_key: "463c0dc01f7045de141de739ce83c6a4"
server_key_iv: "a34f373b0275ed66eb3c524e13c85077"
server_key_version: "encryptionkey@0"
I20260808 04:50:37.871402 20007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:37.895280 20007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:37.898682 20007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:37.898885 20007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:37.899499 20007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:37.900413 20007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:37.900485 20007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.900556 20007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:37.900600 20007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:37.909721 20007 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:39963
I20260808 04:50:37.909739 20212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:39963 every 8 connection(s)
I20260808 04:50:37.910588 20007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:37.916020 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 20007
I20260808 04:50:37.916211 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:37.916592 15481 external_mini_cluster.cc:1428] Setting key 6c1627ea355a6ff43e37cd13e4a9ec8e
Aug 08 04:50:37 dist-test-slave-zpvx krb5kdc[19408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:37.924515 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:37.912473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:43463 (local address 127.15.30.126:46591)
0808 04:50:37.912617 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:37.912621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:37.913440 (+ 819us) server_negotiation.cc:407] Connection header received
0808 04:50:37.914774 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:37.914777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:37.914838 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:37.914946 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:37.916289 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.917092 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.918416 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.918768 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.918886 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:37.921258 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:37.921261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:37.921263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:37.921287 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:37.922463 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.922989 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.922993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.922995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.923078 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:37.923492 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:37.923494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:37.923495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:37.923677 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:37.923773 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:37.924205 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 04:50:37.924320 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:50:37.925262 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:37.912840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:46591 (local address 127.15.30.67:43463)
0808 04:50:37.913294 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:37.913336 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:37.914551 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:37.915127 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:37.915133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:37.915476 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:37.916088 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:37.916113 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.917229 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.917232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:37.918296 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:37.918307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:37.918919 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:37.918923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:37.919106 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:37.919582 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:37.919596 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:37.921076 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:37.922585 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:37.922591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:37.922604 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:37.922880 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:37.923196 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:37.923199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:37.923201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:37.923339 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:37.923785 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:37.923790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:37.923997 (+ 207us) client_negotiation.cc:769] Sending connection context
0808 04:50:37.924150 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 04:50:37.924479 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260808 04:50:37.926386 20215 heartbeater.cc:347] Connected to a master server at 127.15.30.126:46591
I20260808 04:50:37.926615 20215 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:37.927090 20215 heartbeater.cc:511] Master 127.15.30.126:46591 requested a full tablet report, sending...
I20260808 04:50:37.928164 19482 ts_manager.cc:194] Registered new tserver with Master: 88de0e6b22904c2fb5bd036774ba87e3 (127.15.30.67:39963)
I20260808 04:50:37.928869 19482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:43463
I20260808 04:50:37.930558 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:50:37.947405 19549 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:37.945874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.15.30.126:46591)
0808 04:50:37.946036 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:37.946041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:37.946201 (+ 160us) server_negotiation.cc:407] Connection header received
0808 04:50:37.946368 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:37.946378 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:37.946419 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:37.946507 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:37.947268 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53976: BlockingRecv error: recv got EOF from 127.0.0.1:53976 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:50:37.948559 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 19587
I20260808 04:50:37.967049 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 19804
I20260808 04:50:37.979481 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 20007
I20260808 04:50:37.988699 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 19431
2026-08-08T04:50:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2400 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:38 dist-test-slave-zpvx krb5kdc[20233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:38 dist-test-slave-zpvx krb5kdc[20233](info): set up 2 sockets
Aug 08 04:50:38 dist-test-slave-zpvx krb5kdc[20233](info): commencing operation
krb5kdc: starting...
W20260808 04:50:39.247123 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:39 dist-test-slave-zpvx krb5kdc[20233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:39.369906 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43281
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:43429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:39.482808 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:39.483103 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:39.483188 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:39.487283 20253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:39.487368 20253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:39.487406 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:39.487464 20253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:39.487504 20253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:39.492660 20253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:39.493666 20253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:39.494678 20253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:39.503136 20268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:39.503324 20253 server_base.cc:1034] running on GCE node
W20260808 04:50:39.503141 20269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:39.503404 20276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:39.504166 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:39.505295 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:39.506521 20253 hybrid_clock.cc:648] HybridClock initialized: now 1786164639506494 us; error 32 us; skew 500 ppm
Aug 08 04:50:39 dist-test-slave-zpvx krb5kdc[20233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164639, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:39.509467 20253 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:39.510322 20253 webserver.cc:460] Webserver started at http://127.15.30.126:36233/ using document root <none> and password file <none>
I20260808 04:50:39.510874 20253 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:39.510964 20253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:39.511211 20253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:39.513275 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "c082f9c1cae2468395b13af189070b02"
format_stamp: "Formatted at 2026-08-08 04:50:39 on dist-test-slave-zpvx"
server_key: "da852ab98c5b56668ffdaf8629e0e1a6"
server_key_iv: "7bb0bbe5501132cbca212c6f8cba427b"
server_key_version: "encryptionkey@0"
I20260808 04:50:39.513841 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "c082f9c1cae2468395b13af189070b02"
format_stamp: "Formatted at 2026-08-08 04:50:39 on dist-test-slave-zpvx"
server_key: "da852ab98c5b56668ffdaf8629e0e1a6"
server_key_iv: "7bb0bbe5501132cbca212c6f8cba427b"
server_key_version: "encryptionkey@0"
I20260808 04:50:39.517457 20253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:50:39.519842 20286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:39.521056 20253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:39.521203 20253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "c082f9c1cae2468395b13af189070b02"
format_stamp: "Formatted at 2026-08-08 04:50:39 on dist-test-slave-zpvx"
server_key: "da852ab98c5b56668ffdaf8629e0e1a6"
server_key_iv: "7bb0bbe5501132cbca212c6f8cba427b"
server_key_version: "encryptionkey@0"
I20260808 04:50:39.521312 20253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:39.537485 20253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:39.540464 20253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:39.540678 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:39.547972 20253 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:43281
I20260808 04:50:39.548323 20371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:43281 every 8 connection(s)
I20260808 04:50:39.548960 20253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:39.551949 20372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:39.556090 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 20253
I20260808 04:50:39.556216 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:39.556527 15481 external_mini_cluster.cc:1428] Setting key f0af0093a6717c4ca5d785ac03cacb8c
I20260808 04:50:39.558115 20372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02: Bootstrap starting.
I20260808 04:50:39.560650 20372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:39.561395 20372 log.cc:824] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:39.563443 20372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02: No bootstrap required, opened a new log
Aug 08 04:50:39 dist-test-slave-zpvx krb5kdc[20233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:39.568014 20372 raft_consensus.cc:348] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082f9c1cae2468395b13af189070b02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43281 } }
I20260808 04:50:39.568308 20372 raft_consensus.cc:374] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:39.568388 20372 raft_consensus.cc:729] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c082f9c1cae2468395b13af189070b02, State: Initialized, Role: FOLLOWER
I20260808 04:50:39.568862 20372 consensus_queue.cc:260] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082f9c1cae2468395b13af189070b02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43281 } }
I20260808 04:50:39.569046 20372 raft_consensus.cc:388] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:39.569128 20372 raft_consensus.cc:482] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:39.569231 20372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:39.570355 20372 raft_consensus.cc:504] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082f9c1cae2468395b13af189070b02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43281 } }
I20260808 04:50:39.570705 20372 leader_election.cc:302] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c082f9c1cae2468395b13af189070b02; no voters:
I20260808 04:50:39.571036 20372 leader_election.cc:288] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:39.571146 20380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:39.571414 20380 raft_consensus.cc:686] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [term 1 LEADER]: Becoming Leader. State: Replica: c082f9c1cae2468395b13af189070b02, State: Running, Role: LEADER
I20260808 04:50:39.571465 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:39.557843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.15.30.126:43281)
0808 04:50:39.558281 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:39.558297 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:39.558335 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:50:39.558960 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:39.558977 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:39.559314 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:39.559596 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:39.560559 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:39.561648 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:39.562759 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:39.563223 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:39.563345 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:39.566135 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:39.566145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:39.566163 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:39.566194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:39.568151 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:39.568727 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:39.568733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:39.568739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:39.568878 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:39.569223 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:39.569227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:39.569230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:39.569742 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:39.569944 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:39.570180 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 04:50:39.570482 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":142,"threads_started":1}
I20260808 04:50:39.572116 20380 consensus_queue.cc:237] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082f9c1cae2468395b13af189070b02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43281 } }
I20260808 04:50:39.572247 20372 sys_catalog.cc:564] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:39.573959 20382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c082f9c1cae2468395b13af189070b02. Latest consensus state: current_term: 1 leader_uuid: "c082f9c1cae2468395b13af189070b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082f9c1cae2468395b13af189070b02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43281 } } }
I20260808 04:50:39.574115 20382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:39.574517 20383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c082f9c1cae2468395b13af189070b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082f9c1cae2468395b13af189070b02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43281 } } }
I20260808 04:50:39.574698 20383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:39.574851 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 04:50:39.576987 20403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:50:39.577076 20403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:50:39.577196 20401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:39.577916 20401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:39.583969 20401 catalog_manager.cc:1347] Generated new cluster ID: a6c34cf287e84ae09eab2b5c2f28df33
I20260808 04:50:39.584033 20401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:39.591435 20401 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:39.592624 20401 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:39.595841 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:39.590627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.15.30.126:43281)
0808 04:50:39.590745 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:39.590749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:39.591294 (+ 545us) server_negotiation.cc:407] Connection header received
0808 04:50:39.591464 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:39.591469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:39.591530 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:39.591603 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:39.592190 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:39.592908 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:39.593760 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:39.594008 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:39.594132 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:39.594965 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:39.594969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:39.594972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:39.595383 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:39.595410 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:39.595426 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:39.595529 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:39.595619 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 04:50:39.595692 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":47}
I20260808 04:50:39.600500 20401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c082f9c1cae2468395b13af189070b02: Generated new TSK 0
I20260808 04:50:39.601233 20401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:50:39.612284 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 20253
2026-08-08T04:50:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1629 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:39 dist-test-slave-zpvx krb5kdc[20423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:39 dist-test-slave-zpvx krb5kdc[20423](info): set up 2 sockets
Aug 08 04:50:39 dist-test-slave-zpvx krb5kdc[20423](info): commencing operation
krb5kdc: starting...
W20260808 04:50:40.887822 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:40 dist-test-slave-zpvx krb5kdc[20423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:41.011057 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34159
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:34159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:41.124770 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:41.125023 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:41.125064 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:41.129136 20439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:41.129194 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:41.129213 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:41.129230 20439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:41.129245 20439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:41.134373 20439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:34159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:41.135411 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:41.136583 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:41.145437 20449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:41.145422 20452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:41.145442 20448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:41.145788 20439 server_base.cc:1034] running on GCE node
I20260808 04:50:41.146584 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:41.147675 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:41.148880 20439 hybrid_clock.cc:648] HybridClock initialized: now 1786164641148858 us; error 32 us; skew 500 ppm
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:41.151811 20439 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:41.152832 20439 webserver.cc:460] Webserver started at http://127.15.30.126:41537/ using document root <none> and password file <none>
I20260808 04:50:41.153407 20439 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:41.153501 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:41.153743 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:41.155820 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "7ebbcd0fd8a747be90e131017fb6b56e"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "35088669553efbf3f4a75ec4c01cf1ee"
server_key_iv: "24e91b308a2fa5721b8e2f9ab1efeab8"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.156471 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "7ebbcd0fd8a747be90e131017fb6b56e"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "35088669553efbf3f4a75ec4c01cf1ee"
server_key_iv: "24e91b308a2fa5721b8e2f9ab1efeab8"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.160212 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:41.162827 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.164158 20439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:41.164319 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "7ebbcd0fd8a747be90e131017fb6b56e"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "35088669553efbf3f4a75ec4c01cf1ee"
server_key_iv: "24e91b308a2fa5721b8e2f9ab1efeab8"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.164492 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:41.181421 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:41.184733 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:41.184943 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:41.192313 20439 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:34159
I20260808 04:50:41.192632 20545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:34159 every 8 connection(s)
I20260808 04:50:41.193347 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:41.196141 20546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:41.197605 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 20439
I20260808 04:50:41.197747 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:41.198067 15481 external_mini_cluster.cc:1428] Setting key 1f22ac437f14d1d9de8d74eeea36dbc4
I20260808 04:50:41.203030 20546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e: Bootstrap starting.
I20260808 04:50:41.205718 20546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:41.206480 20546 log.cc:824] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:41.208878 20546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e: No bootstrap required, opened a new log
I20260808 04:50:41.213735 20546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34159 } }
I20260808 04:50:41.213999 20546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:41.214057 20546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebbcd0fd8a747be90e131017fb6b56e, State: Initialized, Role: FOLLOWER
I20260808 04:50:41.214099 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.199574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.15.30.126:34159)
0808 04:50:41.200007 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:41.200026 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:41.200070 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:50:41.200926 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:41.200954 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:41.201389 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:41.201761 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:41.202482 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.203911 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.204959 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.205642 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.205818 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.208842 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:41.208848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:41.208865 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:41.208906 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:41.210867 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.211333 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.211341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.211349 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.211514 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.211783 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.211787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.211789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.212294 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:41.212528 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:41.212791 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 04:50:41.213117 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260808 04:50:41.214540 20546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34159 } }
I20260808 04:50:41.214709 20546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:41.214800 20546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:41.214901 20546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:41.216010 20546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34159 } }
I20260808 04:50:41.216424 20546 leader_election.cc:302] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ebbcd0fd8a747be90e131017fb6b56e; no voters:
I20260808 04:50:41.216807 20546 leader_election.cc:288] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:41.216899 20551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:41.217108 20551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [term 1 LEADER]: Becoming Leader. State: Replica: 7ebbcd0fd8a747be90e131017fb6b56e, State: Running, Role: LEADER
I20260808 04:50:41.217445 20551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34159 } }
I20260808 04:50:41.217957 20546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:41.219208 20552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34159 } } }
I20260808 04:50:41.219336 20552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:41.219293 20553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ebbcd0fd8a747be90e131017fb6b56e. Latest consensus state: current_term: 1 leader_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ebbcd0fd8a747be90e131017fb6b56e" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34159 } } }
I20260808 04:50:41.219400 20553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:41.220249 20565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:41.222884 20565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:41.228858 20565 catalog_manager.cc:1347] Generated new cluster ID: aeb01a90118745db87911f2ad6a28837
I20260808 04:50:41.228920 20565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:41.237716 20565 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:41.238579 20565 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:41.247473 20565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ebbcd0fd8a747be90e131017fb6b56e: Generated new TSK 0
I20260808 04:50:41.248243 20565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:41.281629 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34159
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:41.391964 20587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:41.392234 20587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:41.392318 20587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:41.396188 20587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:41.396267 20587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:41.396354 20587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:41.401340 20587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:41.402276 20587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:41.403246 20587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:41.412523 20593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:41.412634 20587 server_base.cc:1034] running on GCE node
W20260808 04:50:41.412520 20595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:41.412554 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:41.413229 20587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:41.413870 20587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:41.415069 20587 hybrid_clock.cc:648] HybridClock initialized: now 1786164641415056 us; error 29 us; skew 500 ppm
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:41.418215 20587 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:41.419240 20587 webserver.cc:460] Webserver started at http://127.15.30.65:39457/ using document root <none> and password file <none>
I20260808 04:50:41.419862 20587 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:41.419955 20587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:41.420174 20587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:41.422219 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "c6d804fa3f4b4316a47cf87492cd2530"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "f6d0bdb35b403a92c7297f171bba446b"
server_key_iv: "92ec9f46f48f0dcb39ad1c91e4ea272c"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.422808 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "c6d804fa3f4b4316a47cf87492cd2530"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "f6d0bdb35b403a92c7297f171bba446b"
server_key_iv: "92ec9f46f48f0dcb39ad1c91e4ea272c"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.426414 20587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:50:41.428758 20608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.429931 20587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:41.430068 20587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "c6d804fa3f4b4316a47cf87492cd2530"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "f6d0bdb35b403a92c7297f171bba446b"
server_key_iv: "92ec9f46f48f0dcb39ad1c91e4ea272c"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.430177 20587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:41.463505 20587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:41.466732 20587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:41.466946 20587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:41.467618 20587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:41.468639 20587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:41.468721 20587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.468789 20587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:41.468834 20587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.478217 20587 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:36871
I20260808 04:50:41.478286 20798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:36871 every 8 connection(s)
I20260808 04:50:41.479175 20587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:41.488308 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 20587
I20260808 04:50:41.488502 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:41.488808 15481 external_mini_cluster.cc:1428] Setting key dcfa9799716a10b8ed03553d31906e41
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:41.494524 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.481197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:40391 (local address 127.15.30.126:34159)
0808 04:50:41.481353 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:41.481358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:41.482248 (+ 890us) server_negotiation.cc:407] Connection header received
0808 04:50:41.483688 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:41.483691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:41.483754 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:41.483838 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:41.485167 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.485717 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.486884 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.487132 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.487248 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.490100 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:41.490103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:41.490105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:41.490132 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:41.492206 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.492896 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.492900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.492901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.492985 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.493346 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.493350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.493351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.493538 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:41.493664 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:41.494212 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 04:50:41.494346 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:50:41.495411 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.481559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34159 (local address 127.15.30.65:40391)
0808 04:50:41.482067 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:41.482123 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:41.483451 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:41.484020 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:41.484027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:41.484467 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:41.485001 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:41.485015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.485853 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.485859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:41.486746 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:41.486756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.487295 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.487299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:41.487479 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.488069 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:41.488089 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:41.489868 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:41.492350 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:41.492355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:41.492422 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:41.492765 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:41.493084 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:41.493088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:41.493090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:41.493235 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:41.493662 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:41.493669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:41.493915 (+ 246us) client_negotiation.cc:769] Sending connection context
0808 04:50:41.494157 (+ 242us) client_negotiation.cc:240] Negotiation successful
0808 04:50:41.494492 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":123,"threads_started":1}
I20260808 04:50:41.496964 20800 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34159
I20260808 04:50:41.497236 20800 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:41.497756 20800 heartbeater.cc:511] Master 127.15.30.126:34159 requested a full tablet report, sending...
I20260808 04:50:41.499370 20488 ts_manager.cc:194] Registered new tserver with Master: c6d804fa3f4b4316a47cf87492cd2530 (127.15.30.65:36871)
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:50:41.500619 20488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:40391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:41.520310 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34159
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:41.632674 20807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:41.632925 20807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:41.633008 20807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:41.637195 20807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:41.637243 20807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:41.637298 20807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:41.642356 20807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:41.643249 20807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:41.644186 20807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:41.653805 20815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:41.653800 20817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:41.653825 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:41.654201 20807 server_base.cc:1034] running on GCE node
I20260808 04:50:41.654621 20807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:41.655260 20807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:41.656484 20807 hybrid_clock.cc:648] HybridClock initialized: now 1786164641656447 us; error 43 us; skew 500 ppm
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:41.659258 20807 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:41.660142 20807 webserver.cc:460] Webserver started at http://127.15.30.66:46401/ using document root <none> and password file <none>
I20260808 04:50:41.660748 20807 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:41.660812 20807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:41.661020 20807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:41.663107 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "cc532bc3435f4e8f8eeb137b84b2bf4c"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "031f2f534243f1ef8940314c8fa982ca"
server_key_iv: "38b086349b1613e3fd6c79af26f9dafc"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.663671 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "cc532bc3435f4e8f8eeb137b84b2bf4c"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "031f2f534243f1ef8940314c8fa982ca"
server_key_iv: "38b086349b1613e3fd6c79af26f9dafc"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.667438 20807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:41.670552 20830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.671846 20807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:41.671985 20807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "cc532bc3435f4e8f8eeb137b84b2bf4c"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "031f2f534243f1ef8940314c8fa982ca"
server_key_iv: "38b086349b1613e3fd6c79af26f9dafc"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.672132 20807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:41.702925 20807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:41.706236 20807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:41.706455 20807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:41.707129 20807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:41.708050 20807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:41.708113 20807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.708189 20807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:41.708230 20807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.718101 20807 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:38145
I20260808 04:50:41.718129 21006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:38145 every 8 connection(s)
I20260808 04:50:41.719257 20807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:41.727798 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 20807
I20260808 04:50:41.727934 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:41.728272 15481 external_mini_cluster.cc:1428] Setting key 293505796869dbc5a36a1b66a583a8e0
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:41.734977 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.721589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:53247 (local address 127.15.30.126:34159)
0808 04:50:41.721752 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:41.721757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:41.722617 (+ 860us) server_negotiation.cc:407] Connection header received
0808 04:50:41.724150 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:41.724154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:41.724217 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:41.724306 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:41.725634 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.726219 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.727496 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.727757 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.727887 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.730515 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:41.730522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:41.730529 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:41.730568 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:41.732502 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.733249 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.733252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.733254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.733380 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.733812 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.733815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.733816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.734002 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:41.734129 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:41.734745 (+ 616us) server_negotiation.cc:299] Negotiation successful
0808 04:50:41.734857 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:50:41.736300 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.721993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34159 (local address 127.15.30.66:53247)
0808 04:50:41.722448 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:41.722493 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:41.723904 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:41.724515 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:41.724523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:41.724904 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:41.725456 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:41.725470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.726369 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.726373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:41.727352 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:41.727362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.727935 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.727940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:41.728138 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.728686 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:41.728704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:41.730339 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:41.732663 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:41.732671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:41.732687 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:41.733154 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:41.733481 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:41.733486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:41.733490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:41.733699 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:41.734122 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:41.734131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:41.734449 (+ 318us) client_negotiation.cc:769] Sending connection context
0808 04:50:41.734671 (+ 222us) client_negotiation.cc:240] Negotiation successful
0808 04:50:41.735047 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260808 04:50:41.737663 21007 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34159
I20260808 04:50:41.737931 21007 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:41.738505 21007 heartbeater.cc:511] Master 127.15.30.126:34159 requested a full tablet report, sending...
I20260808 04:50:41.739660 20488 ts_manager.cc:194] Registered new tserver with Master: cc532bc3435f4e8f8eeb137b84b2bf4c (127.15.30.66:38145)
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:50:41.740258 20488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:53247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:41.759291 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34159
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:41.874699 21018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:41.874948 21018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:41.875026 21018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:41.879130 21018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:41.879218 21018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:41.879314 21018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:41.884531 21018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:41.885562 21018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:41.886600 21018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:41.896251 21030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:41.896266 21027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:41.896309 21018 server_base.cc:1034] running on GCE node
W20260808 04:50:41.896245 21028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:41.896839 21018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:41.897473 21018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:41.898659 21018 hybrid_clock.cc:648] HybridClock initialized: now 1786164641898644 us; error 28 us; skew 500 ppm
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:41.901472 21018 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:41.902316 21018 webserver.cc:460] Webserver started at http://127.15.30.67:43169/ using document root <none> and password file <none>
I20260808 04:50:41.902891 21018 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:41.902985 21018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:41.903231 21018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:41.905429 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "d723cbc84ad04108a562f879deb0c5cf"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "aa1e7bdf9f39632cc1b5d225b0360375"
server_key_iv: "a7ed734f9ae4e197487806eea739aee8"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.906021 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "d723cbc84ad04108a562f879deb0c5cf"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "aa1e7bdf9f39632cc1b5d225b0360375"
server_key_iv: "a7ed734f9ae4e197487806eea739aee8"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.909860 21018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:50:41.912317 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.913595 21018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:41.913738 21018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "d723cbc84ad04108a562f879deb0c5cf"
format_stamp: "Formatted at 2026-08-08 04:50:41 on dist-test-slave-zpvx"
server_key: "aa1e7bdf9f39632cc1b5d225b0360375"
server_key_iv: "a7ed734f9ae4e197487806eea739aee8"
server_key_version: "encryptionkey@0"
I20260808 04:50:41.913849 21018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:41.929365 21018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:41.932005 21018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:41.932204 21018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:41.932869 21018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:41.933756 21018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:41.933835 21018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.933897 21018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:41.933949 21018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:41.943082 21018 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:41511
I20260808 04:50:41.943114 21210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:41511 every 8 connection(s)
I20260808 04:50:41.944013 21018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:41.946252 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 21018
I20260808 04:50:41.946386 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:41.946712 15481 external_mini_cluster.cc:1428] Setting key 803451f5b5134906eb9ff80f9a1c295f
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:41.958421 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.946088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:37085 (local address 127.15.30.126:34159)
0808 04:50:41.946234 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:41.946238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:41.947477 (+ 1239us) server_negotiation.cc:407] Connection header received
0808 04:50:41.949147 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:41.949150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:41.949206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:41.949286 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:41.950492 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.951036 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.952205 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.952547 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.952674 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.955056 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:41.955059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:41.955061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:41.955086 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:41.956604 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.957075 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.957078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.957079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.957177 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.957483 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.957486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.957487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.957652 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:41.957728 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:41.958202 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 04:50:41.958298 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:50:41.959250 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.946620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34159 (local address 127.15.30.67:37085)
0808 04:50:41.947304 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:41.947362 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:41.948878 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:41.949459 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:41.949467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:41.949843 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:41.950344 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:41.950358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.951181 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.951184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:41.952067 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:41.952076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.952708 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.952712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:41.952889 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.953414 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:41.953429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:41.954900 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:41.956718 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:41.956723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:41.956735 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:41.956989 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:41.957262 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:41.957265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:41.957266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:41.957401 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:41.957753 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:41.957758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:41.957973 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 04:50:41.958129 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 04:50:41.958383 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":164,"threads_started":1}
I20260808 04:50:41.960458 21216 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34159
I20260808 04:50:41.960716 21216 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:41.961232 21216 heartbeater.cc:511] Master 127.15.30.126:34159 requested a full tablet report, sending...
I20260808 04:50:41.962278 20487 ts_manager.cc:194] Registered new tserver with Master: d723cbc84ad04108a562f879deb0c5cf (127.15.30.67:41511)
I20260808 04:50:41.962813 20487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:37085
I20260808 04:50:41.972707 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 04:50:41 dist-test-slave-zpvx krb5kdc[20423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164641, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:41.999682 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:41.991201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.15.30.126:34159)
0808 04:50:41.991341 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:41.991345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:41.991502 (+ 157us) server_negotiation.cc:407] Connection header received
0808 04:50:41.991610 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:41.991612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:41.991663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:41.991737 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:41.992465 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.992979 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.993791 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:41.993995 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:41.994072 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:41.996520 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:41.996523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:41.996525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:41.996549 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:41.997840 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.998386 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.998391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.998393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.998479 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:41.998774 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:41.998778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:41.998780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:41.999002 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:41.999145 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:41.999392 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 04:50:41.999511 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 04:50:42.000058 20487 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33882
I20260808 04:50:42.000999 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 20587
I20260808 04:50:42.022675 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 20807
I20260808 04:50:42.041736 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 21018
I20260808 04:50:42.053489 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 20439
2026-08-08T04:50:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2439 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:42 dist-test-slave-zpvx krb5kdc[21243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:42 dist-test-slave-zpvx krb5kdc[21243](info): set up 2 sockets
Aug 08 04:50:42 dist-test-slave-zpvx krb5kdc[21243](info): commencing operation
krb5kdc: starting...
W20260808 04:50:43.307715 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:43 dist-test-slave-zpvx krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:43.435249 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33231
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:43.545750 21261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:43.546020 21261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:43.546093 21261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:43.549984 21261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:43.550067 21261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:43.550107 21261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:43.550143 21261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:43.550174 21261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:43.555173 21261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:43.556131 21261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:43.557123 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:43.565492 21269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:43.565524 21274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:43.565510 21270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:43.565560 21261 server_base.cc:1034] running on GCE node
I20260808 04:50:43.566223 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:43.567168 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:43.568332 21261 hybrid_clock.cc:648] HybridClock initialized: now 1786164643568315 us; error 31 us; skew 500 ppm
Aug 08 04:50:43 dist-test-slave-zpvx krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164643, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:43.576602 21261 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:43.577639 21261 webserver.cc:460] Webserver started at http://127.15.30.126:34893/ using document root <none> and password file <none>
I20260808 04:50:43.578258 21261 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:43.578368 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:43.578624 21261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:43.580937 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "ea94f0a179e042f5ba071772f19cbb02"
format_stamp: "Formatted at 2026-08-08 04:50:43 on dist-test-slave-zpvx"
server_key: "7cc907e16dc07b7f6f8e845344ef000a"
server_key_iv: "17d3689b5e05c13754a2139398d581b1"
server_key_version: "encryptionkey@0"
I20260808 04:50:43.581581 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "ea94f0a179e042f5ba071772f19cbb02"
format_stamp: "Formatted at 2026-08-08 04:50:43 on dist-test-slave-zpvx"
server_key: "7cc907e16dc07b7f6f8e845344ef000a"
server_key_iv: "17d3689b5e05c13754a2139398d581b1"
server_key_version: "encryptionkey@0"
I20260808 04:50:43.585546 21261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:50:43.588014 21284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:43.589313 21261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:43.589465 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "ea94f0a179e042f5ba071772f19cbb02"
format_stamp: "Formatted at 2026-08-08 04:50:43 on dist-test-slave-zpvx"
server_key: "7cc907e16dc07b7f6f8e845344ef000a"
server_key_iv: "17d3689b5e05c13754a2139398d581b1"
server_key_version: "encryptionkey@0"
I20260808 04:50:43.589581 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:43.614596 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:43.618096 21261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:43.618326 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:43.626353 21261 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33231
I20260808 04:50:43.626384 21362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33231 every 8 connection(s)
I20260808 04:50:43.627342 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:43.630101 21363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:43.631968 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 21261
I20260808 04:50:43.632103 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:43.632432 15481 external_mini_cluster.cc:1428] Setting key 56e32dcb47ea515545a4ae796ec52a20
I20260808 04:50:43.636718 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02: Bootstrap starting.
I20260808 04:50:43.638849 21363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:43.639554 21363 log.cc:824] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:43.641538 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02: No bootstrap required, opened a new log
Aug 08 04:50:43 dist-test-slave-zpvx krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:43.645093 21363 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea94f0a179e042f5ba071772f19cbb02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33231 } }
I20260808 04:50:43.645366 21363 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:43.645442 21363 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea94f0a179e042f5ba071772f19cbb02, State: Initialized, Role: FOLLOWER
I20260808 04:50:43.645921 21363 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea94f0a179e042f5ba071772f19cbb02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33231 } }
I20260808 04:50:43.646068 21363 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:43.646154 21363 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:43.646250 21363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:43.647293 21363 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea94f0a179e042f5ba071772f19cbb02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33231 } }
I20260808 04:50:43.647629 21363 leader_election.cc:302] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea94f0a179e042f5ba071772f19cbb02; no voters:
I20260808 04:50:43.647922 21363 leader_election.cc:288] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:43.648047 21368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:43.648270 21368 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [term 1 LEADER]: Becoming Leader. State: Replica: ea94f0a179e042f5ba071772f19cbb02, State: Running, Role: LEADER
I20260808 04:50:43.648427 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:43.633912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.15.30.126:33231)
0808 04:50:43.634367 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:43.634382 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:43.634419 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:50:43.635196 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:43.635222 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:43.635589 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:43.635940 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:43.636865 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:43.637910 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:43.638875 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:43.639444 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:43.639626 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:43.642554 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:43.642560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:43.642573 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:43.642600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:43.644875 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:43.645421 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:43.645445 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:43.645454 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:43.645615 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:43.645937 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:43.645939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:43.645941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:43.646518 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:43.646689 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:43.646971 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 04:50:43.647291 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260808 04:50:43.648702 21368 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea94f0a179e042f5ba071772f19cbb02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33231 } }
I20260808 04:50:43.649199 21363 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:43.649956 21368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea94f0a179e042f5ba071772f19cbb02. Latest consensus state: current_term: 1 leader_uuid: "ea94f0a179e042f5ba071772f19cbb02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea94f0a179e042f5ba071772f19cbb02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33231 } } }
I20260808 04:50:43.650159 21368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:43.651337 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea94f0a179e042f5ba071772f19cbb02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea94f0a179e042f5ba071772f19cbb02" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33231 } } }
I20260808 04:50:43.651795 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:43.651453 21385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:43.654326 21385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:43.660005 21385 catalog_manager.cc:1347] Generated new cluster ID: e2339245fb4449af83739472f398130c
I20260808 04:50:43.660077 21385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:43.667361 21385 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:43.668187 21385 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:43.674991 21385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea94f0a179e042f5ba071772f19cbb02: Generated new TSK 0
I20260808 04:50:43.675737 21385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:43.711483 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33231
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:43.823544 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:43.823784 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:43.823825 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:43.827852 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:43.827905 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:43.827997 21412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:43.833214 21412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:43.834185 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:43.835165 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:43.844477 21423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:43.844477 21429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:43.844466 21424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:43.844826 21412 server_base.cc:1034] running on GCE node
I20260808 04:50:43.845203 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:43.845808 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:43.846982 21412 hybrid_clock.cc:648] HybridClock initialized: now 1786164643846947 us; error 46 us; skew 500 ppm
Aug 08 04:50:43 dist-test-slave-zpvx krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164643, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:43.850063 21412 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:43.850886 21412 webserver.cc:460] Webserver started at http://127.15.30.65:43663/ using document root <none> and password file <none>
I20260808 04:50:43.851475 21412 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:43.851541 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:43.851751 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:43.853838 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "9468b31c72de487cb53e1e1fc3d7ce35"
format_stamp: "Formatted at 2026-08-08 04:50:43 on dist-test-slave-zpvx"
server_key: "26d93e7796df044a4676854e25621a8b"
server_key_iv: "445f013e409d9ec4201cbae6f58f5fcc"
server_key_version: "encryptionkey@0"
I20260808 04:50:43.854417 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "9468b31c72de487cb53e1e1fc3d7ce35"
format_stamp: "Formatted at 2026-08-08 04:50:43 on dist-test-slave-zpvx"
server_key: "26d93e7796df044a4676854e25621a8b"
server_key_iv: "445f013e409d9ec4201cbae6f58f5fcc"
server_key_version: "encryptionkey@0"
I20260808 04:50:43.858305 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:50:43.860792 21441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:43.862026 21412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:43.862165 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "9468b31c72de487cb53e1e1fc3d7ce35"
format_stamp: "Formatted at 2026-08-08 04:50:43 on dist-test-slave-zpvx"
server_key: "26d93e7796df044a4676854e25621a8b"
server_key_iv: "445f013e409d9ec4201cbae6f58f5fcc"
server_key_version: "encryptionkey@0"
I20260808 04:50:43.862282 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:43.880834 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:43.884025 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:43.884223 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:43.884878 21412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:43.885674 21412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:43.885721 21412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:43.885761 21412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:43.885775 21412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:43.894994 21412 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:41775
I20260808 04:50:43.895025 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:41775 every 8 connection(s)
I20260808 04:50:43.895857 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:43.897981 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 21412
I20260808 04:50:43.898113 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:43.898422 15481 external_mini_cluster.cc:1428] Setting key 0cf3145dbcf52e606c5caf640f4830a1
Aug 08 04:50:43 dist-test-slave-zpvx krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164643, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:50:43.911253 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:43.897616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:38031 (local address 127.15.30.126:33231)
0808 04:50:43.897786 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:43.897790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:43.898747 (+ 957us) server_negotiation.cc:407] Connection header received
0808 04:50:43.900344 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:43.900349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:43.900437 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:43.900539 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:43.901693 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:43.902408 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:43.904154 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:43.904475 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:43.904594 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:43.907512 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:43.907515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:43.907517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:43.907540 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:43.909019 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:43.909550 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:43.909554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:43.909556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:43.909673 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:43.910025 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:43.910030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:43.910032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:43.910222 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:43.910328 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:43.910930 (+ 602us) server_negotiation.cc:299] Negotiation successful
0808 04:50:43.911057 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:50:43.912101 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:43.898057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33231 (local address 127.15.30.65:38031)
0808 04:50:43.898578 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:43.898629 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:43.900130 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:43.900724 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:43.900730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:43.901066 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:43.901539 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:43.901552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:43.902551 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:43.902556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:43.904011 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:43.904027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:43.904661 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:43.904668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:43.904914 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:43.905628 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:43.905654 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:43.907340 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:43.909157 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:43.909162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:43.909178 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:43.909441 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:43.909774 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:43.909777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:43.909780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:43.909925 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:43.910344 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:43.910352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:43.910679 (+ 327us) client_negotiation.cc:769] Sending connection context
0808 04:50:43.910887 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 04:50:43.911209 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:50:43.913508 21619 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33231
I20260808 04:50:43.913762 21619 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:43.914278 21619 heartbeater.cc:511] Master 127.15.30.126:33231 requested a full tablet report, sending...
I20260808 04:50:43.915930 21308 ts_manager.cc:194] Registered new tserver with Master: 9468b31c72de487cb53e1e1fc3d7ce35 (127.15.30.65:41775)
I20260808 04:50:43.917217 21308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:38031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:43.928969 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33231
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:44.038329 21627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:44.038555 21627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:44.038596 21627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:44.042618 21627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:44.042668 21627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:44.042757 21627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:44.047953 21627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:44.049005 21627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:44.050073 21627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:44.060626 21636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:44.060640 21640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:44.060748 21637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:44.060819 21627 server_base.cc:1034] running on GCE node
I20260808 04:50:44.062565 21627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:44.063265 21627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:44.064450 21627 hybrid_clock.cc:648] HybridClock initialized: now 1786164644064432 us; error 44 us; skew 500 ppm
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164644, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:44.067494 21627 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:44.068501 21627 webserver.cc:460] Webserver started at http://127.15.30.66:34617/ using document root <none> and password file <none>
I20260808 04:50:44.069097 21627 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:44.069199 21627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:44.069442 21627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:44.071635 21627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "43d97a5652334f1fb62b8ac7c4fd57b2"
format_stamp: "Formatted at 2026-08-08 04:50:44 on dist-test-slave-zpvx"
server_key: "8509628f7e4dbd272345287c63a9f115"
server_key_iv: "50540b4c0a43498b4c87e830f0cae0db"
server_key_version: "encryptionkey@0"
I20260808 04:50:44.072228 21627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "43d97a5652334f1fb62b8ac7c4fd57b2"
format_stamp: "Formatted at 2026-08-08 04:50:44 on dist-test-slave-zpvx"
server_key: "8509628f7e4dbd272345287c63a9f115"
server_key_iv: "50540b4c0a43498b4c87e830f0cae0db"
server_key_version: "encryptionkey@0"
I20260808 04:50:44.076303 21627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260808 04:50:44.078873 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:44.080185 21627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:50:44.080330 21627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "43d97a5652334f1fb62b8ac7c4fd57b2"
format_stamp: "Formatted at 2026-08-08 04:50:44 on dist-test-slave-zpvx"
server_key: "8509628f7e4dbd272345287c63a9f115"
server_key_iv: "50540b4c0a43498b4c87e830f0cae0db"
server_key_version: "encryptionkey@0"
I20260808 04:50:44.080504 21627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:44.110455 21627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:44.113792 21627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:44.113996 21627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:44.114614 21627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:44.115478 21627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:44.115550 21627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:44.115623 21627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:44.115666 21627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:44.124725 21627 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:45773
I20260808 04:50:44.124765 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:45773 every 8 connection(s)
I20260808 04:50:44.125583 21627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164644, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:44.135259 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 21627
I20260808 04:50:44.135385 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:44.135732 15481 external_mini_cluster.cc:1428] Setting key af2348a55467970d096f02564983db3f
I20260808 04:50:44.139330 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:44.127280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:45429 (local address 127.15.30.126:33231)
0808 04:50:44.127418 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:44.127421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:44.128252 (+ 831us) server_negotiation.cc:407] Connection header received
0808 04:50:44.129742 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:44.129746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:44.129816 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:44.129910 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:44.131057 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.131591 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.132765 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.132991 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.133101 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:44.135623 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:44.135626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:44.135629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:44.135653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:44.137301 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.137814 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.137819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.137821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.137907 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.138258 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.138263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.138265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.138447 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:44.138555 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:44.139039 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 04:50:44.139161 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:50:44.140148 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:44.127660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33231 (local address 127.15.30.66:45429)
0808 04:50:44.128101 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:44.128149 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:44.129507 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:44.130097 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:44.130104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:44.130447 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:44.130901 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:44.130914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.131731 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.131735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:44.132631 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:44.132642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.133160 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.133165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:44.133330 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:44.133847 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:44.133862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:44.135451 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:44.137425 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:44.137431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:44.137444 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:44.137706 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:44.138010 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:44.138013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:44.138015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:44.138154 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:44.138555 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:44.138563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:44.138799 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 04:50:44.138986 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 04:50:44.139303 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260808 04:50:44.141301 21845 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33231
I20260808 04:50:44.141544 21845 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:44.142131 21845 heartbeater.cc:511] Master 127.15.30.126:33231 requested a full tablet report, sending...
I20260808 04:50:44.143282 21308 ts_manager.cc:194] Registered new tserver with Master: 43d97a5652334f1fb62b8ac7c4fd57b2 (127.15.30.66:45773)
I20260808 04:50:44.143908 21308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:45429
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:44.163599 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33231
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:44.275058 21851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:44.275301 21851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:44.275342 21851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:44.279387 21851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:44.279443 21851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:44.279541 21851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:44.284761 21851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:44.285740 21851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:44.286737 21851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:44.296439 21858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:44.296406 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:44.296505 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:44.296657 21851 server_base.cc:1034] running on GCE node
I20260808 04:50:44.297145 21851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:44.297753 21851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:44.298918 21851 hybrid_clock.cc:648] HybridClock initialized: now 1786164644298887 us; error 43 us; skew 500 ppm
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164644, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:44.301740 21851 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:44.302533 21851 webserver.cc:460] Webserver started at http://127.15.30.67:44125/ using document root <none> and password file <none>
I20260808 04:50:44.303071 21851 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:44.303167 21851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:44.303400 21851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:44.305487 21851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "35b573b6140e47c9bd38c574d65bf9f7"
format_stamp: "Formatted at 2026-08-08 04:50:44 on dist-test-slave-zpvx"
server_key: "f56bb2c73de0cdaa4fccbb40c0e23f6e"
server_key_iv: "1bba8f1ee9b85bbee4dbd064698d39f2"
server_key_version: "encryptionkey@0"
I20260808 04:50:44.306046 21851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "35b573b6140e47c9bd38c574d65bf9f7"
format_stamp: "Formatted at 2026-08-08 04:50:44 on dist-test-slave-zpvx"
server_key: "f56bb2c73de0cdaa4fccbb40c0e23f6e"
server_key_iv: "1bba8f1ee9b85bbee4dbd064698d39f2"
server_key_version: "encryptionkey@0"
I20260808 04:50:44.309689 21851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:50:44.311983 21876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:44.313169 21851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:44.313287 21851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "35b573b6140e47c9bd38c574d65bf9f7"
format_stamp: "Formatted at 2026-08-08 04:50:44 on dist-test-slave-zpvx"
server_key: "f56bb2c73de0cdaa4fccbb40c0e23f6e"
server_key_iv: "1bba8f1ee9b85bbee4dbd064698d39f2"
server_key_version: "encryptionkey@0"
I20260808 04:50:44.313388 21851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:44.352448 21851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:44.355243 21851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:44.355444 21851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:44.356037 21851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:44.356894 21851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:44.356943 21851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:44.357012 21851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:44.357053 21851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:44.366230 21851 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:34119
I20260808 04:50:44.366257 22049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:34119 every 8 connection(s)
I20260808 04:50:44.367120 21851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:44.370206 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 21851
I20260808 04:50:44.370312 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:44.370620 15481 external_mini_cluster.cc:1428] Setting key df4198ed17cae78065e6916aeac81544
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164644, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:44.380949 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:44.368881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:50883 (local address 127.15.30.126:33231)
0808 04:50:44.369020 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:44.369024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:44.369877 (+ 853us) server_negotiation.cc:407] Connection header received
0808 04:50:44.371473 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:44.371478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:44.371539 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:44.371643 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:44.373180 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.373724 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.374856 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.375084 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.375211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:44.377568 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:44.377571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:44.377573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:44.377599 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:44.379058 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.379546 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.379549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.379550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.379624 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.379922 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.379924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.379925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.380132 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:44.380212 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:44.380731 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 04:50:44.380827 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:50:44.381760 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:44.369220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33231 (local address 127.15.30.67:50883)
0808 04:50:44.369711 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:44.369754 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:44.371221 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:44.371798 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:44.371804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:44.372317 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:44.373036 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:44.373055 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.373845 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.373848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:44.374725 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:44.374734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.375245 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.375248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:44.375392 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:44.375919 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:44.375935 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:44.377404 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:44.379200 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:44.379205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:44.379216 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:44.379456 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:44.379710 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:44.379712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:44.379714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:44.379841 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:44.380243 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:44.380248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:44.380525 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 04:50:44.380690 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 04:50:44.380924 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
I20260808 04:50:44.382877 22051 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33231
I20260808 04:50:44.383136 22051 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:44.383729 22051 heartbeater.cc:511] Master 127.15.30.126:33231 requested a full tablet report, sending...
I20260808 04:50:44.384881 21308 ts_manager.cc:194] Registered new tserver with Master: 35b573b6140e47c9bd38c574d65bf9f7 (127.15.30.67:34119)
I20260808 04:50:44.385474 21308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:50883
I20260808 04:50:44.396258 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:44.422596 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:44.413803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.15.30.65:41775)
0808 04:50:44.414065 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:44.414072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:44.414104 (+ 32us) server_negotiation.cc:407] Connection header received
0808 04:50:44.414150 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:44.414155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:44.414337 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:44.414454 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:44.415097 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.415840 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.416717 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.417125 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.417215 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:44.419449 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:44.419458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:44.419470 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:44.419494 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:44.421137 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.421521 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.421525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.421527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.421603 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.421836 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.421838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.421840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.422049 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:44.422215 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:44.422354 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 04:50:44.422469 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:44.434255 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:44.425251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.15.30.126:33231)
0808 04:50:44.425441 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:44.425446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:44.425560 (+ 114us) server_negotiation.cc:407] Connection header received
0808 04:50:44.425724 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:44.425730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:44.425781 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:44.425879 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:44.426560 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.427120 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.427968 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:44.428210 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:44.428324 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:44.431100 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:44.431104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:44.431107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:44.431140 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:44.432581 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.432964 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.432967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.432968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.433047 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:44.433360 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:44.433363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:44.433364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:44.433574 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:44.433701 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:44.433924 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 04:50:44.434045 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260808 04:50:44.434716 21308 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36524
I20260808 04:50:44.435398 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 21412
I20260808 04:50:44.445914 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 21627
I20260808 04:50:44.470808 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 21851
I20260808 04:50:44.485445 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 21261
2026-08-08T04:50:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2428 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[22071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[22071](info): set up 2 sockets
Aug 08 04:50:44 dist-test-slave-zpvx krb5kdc[22071](info): commencing operation
krb5kdc: starting...
W20260808 04:50:45.864663 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.352s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260808 04:50:46.161556 15481 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.297s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260808 04:50:46.391304 15481 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.230s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 04:50:46.501624 15481 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.110s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:46 dist-test-slave-zpvx krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:46.563347 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:37685
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:37685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:46.673400 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:46.673641 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:46.673731 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:46.677752 22093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:46.677804 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:46.677824 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:46.677841 22093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:46.677856 22093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:46.682942 22093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:37685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:37685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:46.683941 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:46.685051 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:46.695566 22107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:46.695616 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:46.695604 22093 server_base.cc:1034] running on GCE node
W20260808 04:50:46.695652 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:46.696564 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:46.697759 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:46.698931 22093 hybrid_clock.cc:648] HybridClock initialized: now 1786164646698907 us; error 35 us; skew 500 ppm
Aug 08 04:50:46 dist-test-slave-zpvx krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:46.701973 22093 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:46.702762 22093 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:46.702867 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:46.703200 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:46.706329 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "1c78bc1e7849454b83030c8e4d1b52f9"
format_stamp: "Formatted at 2026-08-08 04:50:46 on dist-test-slave-zpvx"
server_key: "6343d1dd50c64dd612a7cd50916c55d8"
server_key_iv: "51b9b8e926f5056b620bd782696831c5"
server_key_version: "encryptionkey@0"
I20260808 04:50:46.706957 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "1c78bc1e7849454b83030c8e4d1b52f9"
format_stamp: "Formatted at 2026-08-08 04:50:46 on dist-test-slave-zpvx"
server_key: "6343d1dd50c64dd612a7cd50916c55d8"
server_key_iv: "51b9b8e926f5056b620bd782696831c5"
server_key_version: "encryptionkey@0"
I20260808 04:50:46.710923 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:50:46.713552 22117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:46.714804 22093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:46.714951 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "1c78bc1e7849454b83030c8e4d1b52f9"
format_stamp: "Formatted at 2026-08-08 04:50:46 on dist-test-slave-zpvx"
server_key: "6343d1dd50c64dd612a7cd50916c55d8"
server_key_iv: "51b9b8e926f5056b620bd782696831c5"
server_key_version: "encryptionkey@0"
I20260808 04:50:46.715054 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:46.748499 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:46.752162 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:46.752408 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:46.759763 22093 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:37685
I20260808 04:50:46.760053 22201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:37685 every 8 connection(s)
I20260808 04:50:46.760517 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:46.763269 22202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:46.768810 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9: Bootstrap starting.
I20260808 04:50:46.769501 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 22093
I20260808 04:50:46.769647 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:46.769944 15481 external_mini_cluster.cc:1428] Setting key 4969fbf77aec67fc388de77abb467ff2
I20260808 04:50:46.771746 22202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:46.772542 22202 log.cc:824] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:46.774384 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9: No bootstrap required, opened a new log
I20260808 04:50:46.777717 22202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37685 } }
I20260808 04:50:46.777966 22202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:46.778038 22202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c78bc1e7849454b83030c8e4d1b52f9, State: Initialized, Role: FOLLOWER
Aug 08 04:50:46 dist-test-slave-zpvx krb5kdc[22071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:46.778477 22202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37685 } }
I20260808 04:50:46.778657 22202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:46.778743 22202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:46.778842 22202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:46.779665 22202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37685 } }
I20260808 04:50:46.780000 22202 leader_election.cc:302] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c78bc1e7849454b83030c8e4d1b52f9; no voters:
I20260808 04:50:46.780360 22202 leader_election.cc:288] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:46.780445 22208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:46.780658 22208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c78bc1e7849454b83030c8e4d1b52f9, State: Running, Role: LEADER
I20260808 04:50:46.781005 22208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37685 } }
I20260808 04:50:46.781622 22202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:46.782194 22211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c78bc1e7849454b83030c8e4d1b52f9. Latest consensus state: current_term: 1 leader_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37685 } } }
I20260808 04:50:46.782411 22211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:46.782871 22219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:46.785091 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:46.771212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.15.30.126:37685)
0808 04:50:46.771643 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:46.771657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:46.771693 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:50:46.772411 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:46.772440 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:46.772808 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:46.773136 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:46.773986 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:46.774989 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:46.775868 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:46.776339 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:46.776581 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:46.779107 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:46.779112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:46.779125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:46.779151 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:46.781265 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:46.781764 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:46.781769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:46.781775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:46.781895 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:46.782239 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:46.782243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:46.782244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:46.782790 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:46.782970 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:46.783812 (+ 842us) server_negotiation.cc:299] Negotiation successful
0808 04:50:46.784110 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260808 04:50:46.786052 22209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c78bc1e7849454b83030c8e4d1b52f9" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 37685 } } }
I20260808 04:50:46.786226 22219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:46.786231 22209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:46.791882 22219 catalog_manager.cc:1347] Generated new cluster ID: fc697cfa21ce475891e2c9944ef16569
I20260808 04:50:46.791947 22219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:46.808918 22219 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:46.810182 22219 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:46.820295 22219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9: Generated new TSK 0
I20260808 04:50:46.821015 22219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:46.849745 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37685
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:46.958420 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:46.958678 22240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:46.958756 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:46.962689 22240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:46.962769 22240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:46.962859 22240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:46.967803 22240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:46.968858 22240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:46.969818 22240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:46.978838 22249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:46.978863 22248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:46.978864 22252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:46.979108 22240 server_base.cc:1034] running on GCE node
I20260808 04:50:46.979615 22240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:46.980196 22240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:46.981377 22240 hybrid_clock.cc:648] HybridClock initialized: now 1786164646981365 us; error 25 us; skew 500 ppm
Aug 08 04:50:46 dist-test-slave-zpvx krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:46.984338 22240 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:46.985061 22240 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:46.985123 22240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:46.985292 22240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:46.987217 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "e112e7b61daf4e9ebf37cb33657e8bdd"
format_stamp: "Formatted at 2026-08-08 04:50:46 on dist-test-slave-zpvx"
server_key: "593d8fd6f38f8b9cb56821eea3889d58"
server_key_iv: "46f3f233a25112213250eb419632a975"
server_key_version: "encryptionkey@0"
I20260808 04:50:46.987775 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "e112e7b61daf4e9ebf37cb33657e8bdd"
format_stamp: "Formatted at 2026-08-08 04:50:46 on dist-test-slave-zpvx"
server_key: "593d8fd6f38f8b9cb56821eea3889d58"
server_key_iv: "46f3f233a25112213250eb419632a975"
server_key_version: "encryptionkey@0"
I20260808 04:50:46.991290 22240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:46.993604 22261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:46.994854 22240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:46.994972 22240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "e112e7b61daf4e9ebf37cb33657e8bdd"
format_stamp: "Formatted at 2026-08-08 04:50:46 on dist-test-slave-zpvx"
server_key: "593d8fd6f38f8b9cb56821eea3889d58"
server_key_iv: "46f3f233a25112213250eb419632a975"
server_key_version: "encryptionkey@0"
I20260808 04:50:46.995075 22240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:47.019038 22240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:47.022182 22240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:47.022377 22240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:47.022913 22240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:47.023682 22240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:47.023728 22240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.023797 22240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:47.023829 22240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.032649 22240 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:41007
I20260808 04:50:47.032684 22452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:41007 every 8 connection(s)
I20260808 04:50:47.033186 22240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:47.036047 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 22240
I20260808 04:50:47.036193 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:47.036537 15481 external_mini_cluster.cc:1428] Setting key 7317a5fcd9a5a1b69f420bc489a2b772
Aug 08 04:50:47 dist-test-slave-zpvx krb5kdc[22071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:50:47.047485 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.034748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:51005 (local address 127.15.30.126:37685)
0808 04:50:47.034891 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.034896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.036232 (+ 1336us) server_negotiation.cc:407] Connection header received
0808 04:50:47.037362 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.037365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.037411 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.037497 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:47.038881 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.039389 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.040862 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.041112 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.041230 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.043865 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:47.043869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:47.043873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:47.043904 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:47.045435 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.045956 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.045960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.045962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.046046 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.046387 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.046392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.046394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.046573 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:47.046680 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.047160 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.047307 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260808 04:50:47.048195 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.035154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:37685 (local address 127.15.30.65:51005)
0808 04:50:47.036018 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.036076 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.037108 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.037622 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.037628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.038097 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:47.038724 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.038746 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.039516 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.039523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.040740 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.040750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.041268 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.041272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.041440 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.041948 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:47.041973 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:47.043688 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:47.045566 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:47.045572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:47.045586 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:47.045851 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:47.046139 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:47.046143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:47.046145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:47.046289 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:47.046680 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:47.046687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:47.046929 (+ 242us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.047113 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.047418 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260808 04:50:47.049662 22454 heartbeater.cc:347] Connected to a master server at 127.15.30.126:37685
I20260808 04:50:47.049950 22454 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:50:47.050657 22454 heartbeater.cc:511] Master 127.15.30.126:37685 requested a full tablet report, sending...
I20260808 04:50:47.052608 22145 ts_manager.cc:194] Registered new tserver with Master: e112e7b61daf4e9ebf37cb33657e8bdd (127.15.30.65:41007)
I20260808 04:50:47.054267 22145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:51005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:47.071496 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37685
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:47.183871 22466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:47.184090 22466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:47.184131 22466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:47.188088 22466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:47.188143 22466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:47.188246 22466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:47.193281 22466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:47.194216 22466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:47.195165 22466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:47.204542 22475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:47.204542 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:47.204643 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:47.204751 22466 server_base.cc:1034] running on GCE node
I20260808 04:50:47.205186 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:47.205783 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:47.206945 22466 hybrid_clock.cc:648] HybridClock initialized: now 1786164647206932 us; error 26 us; skew 500 ppm
Aug 08 04:50:47 dist-test-slave-zpvx krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:47.209749 22466 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:47.210484 22466 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:47.210549 22466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:47.210762 22466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:47.212780 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "015d29dab37440d9973491e1d6762a59"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "c268ed7085c54bfb5127a809b168a112"
server_key_iv: "cd6bcea2bd5e59bc88dc236d02cfaab2"
server_key_version: "encryptionkey@0"
I20260808 04:50:47.213347 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "015d29dab37440d9973491e1d6762a59"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "c268ed7085c54bfb5127a809b168a112"
server_key_iv: "cd6bcea2bd5e59bc88dc236d02cfaab2"
server_key_version: "encryptionkey@0"
I20260808 04:50:47.217721 22466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:50:47.221112 22486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.222314 22466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260808 04:50:47.222448 22466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "015d29dab37440d9973491e1d6762a59"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "c268ed7085c54bfb5127a809b168a112"
server_key_iv: "cd6bcea2bd5e59bc88dc236d02cfaab2"
server_key_version: "encryptionkey@0"
I20260808 04:50:47.222554 22466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:47.235507 22466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:47.238536 22466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:47.238731 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:47.239378 22466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:47.241834 22466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:47.241948 22466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.242019 22466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:47.242084 22466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.252942 22466 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:43975
I20260808 04:50:47.252961 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:43975 every 8 connection(s)
I20260808 04:50:47.253542 22466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:47.258133 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 22466
I20260808 04:50:47.258258 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:47.258615 15481 external_mini_cluster.cc:1428] Setting key e842c75aafef61d17b0d82239b428b38
Aug 08 04:50:47 dist-test-slave-zpvx krb5kdc[22071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:47.270077 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.255323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:55499 (local address 127.15.30.126:37685)
0808 04:50:47.255459 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.255462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.256319 (+ 857us) server_negotiation.cc:407] Connection header received
0808 04:50:47.257708 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.257711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.257767 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.257874 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:47.259541 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.260173 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.262139 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.262352 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.262439 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.265693 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:47.265700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:47.265701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:47.265727 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:47.267636 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.268362 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.268410 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.268416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.268492 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.268925 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.268927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.268928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.269101 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:47.269175 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.269854 (+ 679us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.269961 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:50:47.271143 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.255717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:37685 (local address 127.15.30.66:55499)
0808 04:50:47.256173 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.256215 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.257502 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.258049 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.258056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.258563 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:47.259346 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.259376 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.260318 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.260321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.261981 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.262000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.262529 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.262536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.262801 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.263435 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:47.263450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:47.265452 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:47.267773 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:47.267782 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:47.267801 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:47.268238 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:47.268591 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:47.268597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:47.268600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:47.268810 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:47.269210 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:47.269219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:47.269532 (+ 313us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.269760 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.270090 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:50:47.272338 22666 heartbeater.cc:347] Connected to a master server at 127.15.30.126:37685
I20260808 04:50:47.272583 22666 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:47.272965 22666 heartbeater.cc:511] Master 127.15.30.126:37685 requested a full tablet report, sending...
I20260808 04:50:47.273939 22145 ts_manager.cc:194] Registered new tserver with Master: 015d29dab37440d9973491e1d6762a59 (127.15.30.66:43975)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
I20260808 04:50:47.274547 22145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:55499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:47.291108 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37685
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:47.400293 22675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:47.400612 22675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:47.400691 22675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:47.404552 22675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:47.404629 22675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:47.404721 22675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:47.409777 22675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:37685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:47.410702 22675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:47.411639 22675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:47.420858 22681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:47.420847 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:47.420964 22675 server_base.cc:1034] running on GCE node
W20260808 04:50:47.420847 22684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:47.421489 22675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:47.422072 22675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:47.423256 22675 hybrid_clock.cc:648] HybridClock initialized: now 1786164647423243 us; error 35 us; skew 500 ppm
Aug 08 04:50:47 dist-test-slave-zpvx krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:47.426185 22675 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:47.426894 22675 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:47.426956 22675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:47.427110 22675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:47.429111 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "0979066182104a0fa0246ddae9f327a4"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "69b3d13a9a5975b844a78eee2788cd65"
server_key_iv: "862ad9e687e5b266b691de8010b139af"
server_key_version: "encryptionkey@0"
I20260808 04:50:47.429666 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "0979066182104a0fa0246ddae9f327a4"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "69b3d13a9a5975b844a78eee2788cd65"
server_key_iv: "862ad9e687e5b266b691de8010b139af"
server_key_version: "encryptionkey@0"
I20260808 04:50:47.433418 22675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:50:47.435806 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.437057 22675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:47.437183 22675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "0979066182104a0fa0246ddae9f327a4"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "69b3d13a9a5975b844a78eee2788cd65"
server_key_iv: "862ad9e687e5b266b691de8010b139af"
server_key_version: "encryptionkey@0"
I20260808 04:50:47.437279 22675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:47.471854 22675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:47.475186 22675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:47.475389 22675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:47.475981 22675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:47.476907 22675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:47.476958 22675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.476997 22675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:47.477012 22675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:47.486163 22675 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:39389
I20260808 04:50:47.486181 22875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:39389 every 8 connection(s)
I20260808 04:50:47.486737 22675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:47.487037 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 22675
I20260808 04:50:47.487195 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:47.487558 15481 external_mini_cluster.cc:1428] Setting key 4399fb10b0735f926e8da4c40da2e74f
Aug 08 04:50:47 dist-test-slave-zpvx krb5kdc[22071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:47.500921 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.488584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:37403 (local address 127.15.30.126:37685)
0808 04:50:47.488719 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.488723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.490140 (+ 1417us) server_negotiation.cc:407] Connection header received
0808 04:50:47.490989 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.490991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.491037 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.491120 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:47.492148 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.492726 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.493987 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.494219 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.494313 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.496824 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:47.496827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:47.496829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:47.496856 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:47.498681 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.499234 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.499237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.499238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.499320 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.499716 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.499719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.499720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.499900 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:47.499977 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.500620 (+ 643us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.500749 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:50:47.501935 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.489094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:37685 (local address 127.15.30.67:37403)
0808 04:50:47.489991 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.490054 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.490809 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.491246 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.491252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.491579 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:47.492021 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.492035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.492823 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.492826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.493802 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.493813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.494384 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.494387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.494541 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.495076 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:47.495092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:47.496661 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:47.498817 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:47.498822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:47.498834 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:47.499130 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:47.499408 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:47.499410 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:47.499412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:47.499637 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:47.499994 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:47.500000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:47.500291 (+ 291us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.500547 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.500870 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":151,"threads_started":1}
I20260808 04:50:47.503131 22876 heartbeater.cc:347] Connected to a master server at 127.15.30.126:37685
I20260808 04:50:47.503376 22876 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:47.503830 22876 heartbeater.cc:511] Master 127.15.30.126:37685 requested a full tablet report, sending...
I20260808 04:50:47.504967 22145 ts_manager.cc:194] Registered new tserver with Master: 0979066182104a0fa0246ddae9f327a4 (127.15.30.67:39389)
I20260808 04:50:47.505549 22145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:37403
I20260808 04:50:47.513777 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:50:47.523226 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.515753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.15.30.126:37685)
0808 04:50:47.515928 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.515932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.516085 (+ 153us) server_negotiation.cc:407] Connection header received
0808 04:50:47.516250 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.516252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.516291 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.516373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:47.517133 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.517637 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.518514 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.518778 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.518891 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.520237 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:47.520241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:47.520245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:47.520274 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:47.521526 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.521960 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.521964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.521966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.522048 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:47.522332 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:47.522336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:47.522338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:47.522560 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:47.522712 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.522950 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.523066 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:50:47.525964 22145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:47.529109 22145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:47.543793 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.537521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.15.30.65:41007)
0808 04:50:47.537902 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.537909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.538552 (+ 643us) server_negotiation.cc:407] Connection header received
0808 04:50:47.538796 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.538803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.538984 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.539149 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:47.539467 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.540630 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.542300 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.543110 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.543233 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.543396 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.543538 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.543635 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":110,"threads_started":1}
I20260808 04:50:47.543798 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.537490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:39389 (local address 127.0.0.1:55250)
0808 04:50:47.538023 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.538038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.538136 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.539351 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.539355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.539373 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:47.539484 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.539491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.540858 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.540862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.542372 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.542382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.543387 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.543390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.543508 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.543542 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.543594 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.543655 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":168,"thread_start_us":105,"threads_started":1}
I20260808 04:50:47.543939 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.538096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:43975 (local address 127.0.0.1:39276)
0808 04:50:47.538346 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.538359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.538629 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.539902 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.539905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.539925 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:47.540004 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.540009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.541370 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.541374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.542497 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.542505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.543465 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.543472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.543612 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.543633 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.543702 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.543761 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":46,"thread_start_us":100,"threads_started":1}
I20260808 04:50:47.543809 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.537399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:41007 (local address 127.0.0.1:57460)
0808 04:50:47.537860 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.537905 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.538611 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.539147 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.539153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.539175 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:47.539325 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.539332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.540745 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.540751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.542172 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.542183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.543255 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.543259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.543362 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.543398 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.543512 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.543627 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":111,"thread_start_us":83,"threads_started":1}
I20260808 04:50:47.544137 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.538262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.15.30.66:43975)
0808 04:50:47.539469 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.539481 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.539504 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:50:47.539575 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.539580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.539760 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.539900 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:47.540119 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.541218 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.542619 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.543132 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.543266 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.543400 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.543871 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.543969 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":135,"threads_started":1}
I20260808 04:50:47.544715 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.538519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.15.30.67:39389)
0808 04:50:47.538990 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.538999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.539018 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:50:47.539077 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.539080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.539219 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.539368 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:47.539610 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.540672 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.542497 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.543250 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.544220 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.544397 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.544459 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.544542 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":167,"threads_started":1}
I20260808 04:50:47.546873 22559 tablet_service.cc:1467] Processing CreateTablet for tablet d0241f24edaa4764911809ce090babdc (DEFAULT_TABLE table=test-table [id=dc675d4c5a7d4bf8b0cd35d49fab5f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:47.547128 22347 tablet_service.cc:1467] Processing CreateTablet for tablet d0241f24edaa4764911809ce090babdc (DEFAULT_TABLE table=test-table [id=dc675d4c5a7d4bf8b0cd35d49fab5f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:47.547945 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0241f24edaa4764911809ce090babdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:47.547713 22767 tablet_service.cc:1467] Processing CreateTablet for tablet d0241f24edaa4764911809ce090babdc (DEFAULT_TABLE table=test-table [id=dc675d4c5a7d4bf8b0cd35d49fab5f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:47.548169 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0241f24edaa4764911809ce090babdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:47.548669 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0241f24edaa4764911809ce090babdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:47.554382 22900 tablet_bootstrap.cc:492] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Bootstrap starting.
I20260808 04:50:47.555701 22901 tablet_bootstrap.cc:492] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd: Bootstrap starting.
I20260808 04:50:47.556092 22902 tablet_bootstrap.cc:492] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Bootstrap starting.
I20260808 04:50:47.556969 22900 tablet_bootstrap.cc:654] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:47.558063 22901 tablet_bootstrap.cc:654] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:47.558159 22900 log.cc:824] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:47.558238 22902 tablet_bootstrap.cc:654] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:47.558873 22901 log.cc:824] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:47.559365 22902 log.cc:824] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:47.560953 22900 tablet_bootstrap.cc:492] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: No bootstrap required, opened a new log
I20260808 04:50:47.560953 22901 tablet_bootstrap.cc:492] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd: No bootstrap required, opened a new log
I20260808 04:50:47.561058 22902 tablet_bootstrap.cc:492] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: No bootstrap required, opened a new log
I20260808 04:50:47.561139 22900 ts_tablet_manager.cc:1397] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:50:47.561138 22901 ts_tablet_manager.cc:1397] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:50:47.561187 22902 ts_tablet_manager.cc:1397] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:50:47.563772 22902 raft_consensus.cc:348] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.563966 22902 raft_consensus.cc:374] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:47.563997 22902 raft_consensus.cc:729] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0979066182104a0fa0246ddae9f327a4, State: Initialized, Role: FOLLOWER
I20260808 04:50:47.564428 22902 consensus_queue.cc:260] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.565071 22876 heartbeater.cc:503] Master 127.15.30.126:37685 was elected leader, sending a full tablet report...
I20260808 04:50:47.564982 22900 raft_consensus.cc:348] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.565228 22900 raft_consensus.cc:374] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:47.565330 22900 raft_consensus.cc:729] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015d29dab37440d9973491e1d6762a59, State: Initialized, Role: FOLLOWER
I20260808 04:50:47.565430 22902 ts_tablet_manager.cc:1428] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 04:50:47.565311 22901 raft_consensus.cc:348] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.565534 22901 raft_consensus.cc:374] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:47.565603 22901 raft_consensus.cc:729] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e112e7b61daf4e9ebf37cb33657e8bdd, State: Initialized, Role: FOLLOWER
I20260808 04:50:47.566030 22901 consensus_queue.cc:260] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.566082 22900 consensus_queue.cc:260] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.566744 22454 heartbeater.cc:503] Master 127.15.30.126:37685 was elected leader, sending a full tablet report...
I20260808 04:50:47.566960 22900 ts_tablet_manager.cc:1428] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:50:47.566960 22901 ts_tablet_manager.cc:1428] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:50:47.567273 22666 heartbeater.cc:503] Master 127.15.30.126:37685 was elected leader, sending a full tablet report...
I20260808 04:50:47.693926 22909 raft_consensus.cc:482] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:47.694288 22909 raft_consensus.cc:504] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.695534 22909 leader_election.cc:288] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e112e7b61daf4e9ebf37cb33657e8bdd (127.15.30.65:41007), 015d29dab37440d9973491e1d6762a59 (127.15.30.66:43975)
I20260808 04:50:47.699546 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.695698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:41007 (local address 127.15.30.67:59557)
0808 04:50:47.695928 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.695945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.696038 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.696364 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.696393 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.696437 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:47.696567 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.696577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.697411 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.697414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.698353 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.698363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.699119 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.699123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.699231 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.699257 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.699338 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.699406 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260808 04:50:47.699585 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.695877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:59557 (local address 127.15.30.65:41007)
0808 04:50:47.696013 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.696018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.696035 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:50:47.696188 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.696193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.696250 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.696334 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:47.696688 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.697285 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.698471 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.699014 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.699146 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.699208 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.699358 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.699471 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:50:47.700022 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.695808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:43975 (local address 127.15.30.67:54885)
0808 04:50:47.696232 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.696247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:47.696350 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:47.696655 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:47.696658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:47.696683 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:47.696797 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.696803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.697640 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.697642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.698717 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:47.698728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.699632 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.699636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:47.699750 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.699780 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:50:47.699854 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:50:47.699899 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260808 04:50:47.700142 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.695930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:54885 (local address 127.15.30.66:43975)
0808 04:50:47.696052 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.696056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.696344 (+ 288us) server_negotiation.cc:407] Connection header received
0808 04:50:47.696478 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.696481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.696533 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.696620 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:47.696942 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.697525 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.698873 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.699491 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.699613 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.699667 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.699892 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.699972 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:50:47.700518 22376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0241f24edaa4764911809ce090babdc" candidate_uuid: "0979066182104a0fa0246ddae9f327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" is_pre_election: true
I20260808 04:50:47.700770 22590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0241f24edaa4764911809ce090babdc" candidate_uuid: "0979066182104a0fa0246ddae9f327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015d29dab37440d9973491e1d6762a59" is_pre_election: true
I20260808 04:50:47.700827 22376 raft_consensus.cc:2454] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0979066182104a0fa0246ddae9f327a4 in term 0.
I20260808 04:50:47.701050 22590 raft_consensus.cc:2454] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0979066182104a0fa0246ddae9f327a4 in term 0.
I20260808 04:50:47.701342 22697 leader_election.cc:302] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0979066182104a0fa0246ddae9f327a4, e112e7b61daf4e9ebf37cb33657e8bdd; no voters:
I20260808 04:50:47.701664 22909 raft_consensus.cc:2781] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:47.701749 22909 raft_consensus.cc:482] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:47.701779 22909 raft_consensus.cc:3037] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:47.702855 22909 raft_consensus.cc:504] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.703234 22909 leader_election.cc:288] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [CANDIDATE]: Term 1 election: Requested vote from peers e112e7b61daf4e9ebf37cb33657e8bdd (127.15.30.65:41007), 015d29dab37440d9973491e1d6762a59 (127.15.30.66:43975)
I20260808 04:50:47.703609 22590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0241f24edaa4764911809ce090babdc" candidate_uuid: "0979066182104a0fa0246ddae9f327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "015d29dab37440d9973491e1d6762a59"
I20260808 04:50:47.703703 22376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0241f24edaa4764911809ce090babdc" candidate_uuid: "0979066182104a0fa0246ddae9f327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd"
I20260808 04:50:47.703805 22376 raft_consensus.cc:3037] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:47.703801 22590 raft_consensus.cc:3037] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:47.705129 22590 raft_consensus.cc:2454] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0979066182104a0fa0246ddae9f327a4 in term 1.
I20260808 04:50:47.705129 22376 raft_consensus.cc:2454] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0979066182104a0fa0246ddae9f327a4 in term 1.
I20260808 04:50:47.705458 22699 leader_election.cc:302] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015d29dab37440d9973491e1d6762a59, 0979066182104a0fa0246ddae9f327a4; no voters:
I20260808 04:50:47.705624 22909 raft_consensus.cc:2781] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:47.705915 22909 raft_consensus.cc:686] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 1 LEADER]: Becoming Leader. State: Replica: 0979066182104a0fa0246ddae9f327a4, State: Running, Role: LEADER
I20260808 04:50:47.706262 22909 consensus_queue.cc:237] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } }
I20260808 04:50:47.710048 22144 catalog_manager.cc:5495] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0979066182104a0fa0246ddae9f327a4 (127.15.30.67). New cstate: current_term: 1 leader_uuid: "0979066182104a0fa0246ddae9f327a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0979066182104a0fa0246ddae9f327a4" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 39389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 } health_report { overall_health: UNKNOWN } } }
I20260808 04:50:47.729068 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.725734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.15.30.67:39389)
0808 04:50:47.725857 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:47.725860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:47.725894 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:50:47.726041 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:47.726044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:47.726093 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:47.726200 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:47.726435 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.726942 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.727754 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:47.727954 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:47.728027 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:47.728278 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:47.728737 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:47.728829 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:47.728911 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 04:50:47.728960 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:50:47.734884 22376 raft_consensus.cc:1261] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd [term 1 FOLLOWER]: Refusing update from remote peer 0979066182104a0fa0246ddae9f327a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:47.734884 22590 raft_consensus.cc:1261] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 1 FOLLOWER]: Refusing update from remote peer 0979066182104a0fa0246ddae9f327a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:47.735821 22913 consensus_queue.cc:1035] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015d29dab37440d9973491e1d6762a59" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:47.736011 22909 consensus_queue.cc:1035] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e112e7b61daf4e9ebf37cb33657e8bdd" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:50:47.737720 22877 tablet.cc:2369] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:47.744086 22919 mvcc.cc:205] Tried to move back new op lower bound from 7316130397114507264 to 7316130397007921152. Current Snapshot: MvccSnapshot[applied={T|T < 7316130397114507264}]
I20260808 04:50:47.744071 22920 mvcc.cc:205] Tried to move back new op lower bound from 7316130397114507264 to 7316130397007921152. Current Snapshot: MvccSnapshot[applied={T|T < 7316130397114507264}]
I20260808 04:50:47.753190 22145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:47.753376 22145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:47.755815 22145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c78bc1e7849454b83030c8e4d1b52f9: Sending DeleteTablet for 3 replicas of tablet d0241f24edaa4764911809ce090babdc
I20260808 04:50:47.756629 22347 tablet_service.cc:1514] Processing DeleteTablet for tablet d0241f24edaa4764911809ce090babdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:47 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:57460
I20260808 04:50:47.756642 22767 tablet_service.cc:1514] Processing DeleteTablet for tablet d0241f24edaa4764911809ce090babdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:47 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:55250
I20260808 04:50:47.757144 22935 tablet_replica.cc:330] T d0241f24edaa4764911809ce090babdc P e112e7b61daf4e9ebf37cb33657e8bdd: stopping tablet replica
I20260808 04:50:47.757154 22559 tablet_service.cc:1514] Processing DeleteTablet for tablet d0241f24edaa4764911809ce090babdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:47 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39276
I20260808 04:50:47.757380 22933 tablet_replica.cc:330] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: stopping tablet replica
I20260808 04:50:47.757684 22933 raft_consensus.cc:2229] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:47.757795 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 22240
I20260808 04:50:47.757910 22938 tablet_replica.cc:330] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: stopping tablet replica
I20260808 04:50:47.758065 22933 raft_consensus.cc:2258] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:47.758210 22938 raft_consensus.cc:2229] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:47.758584 22938 raft_consensus.cc:2258] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:47.773680 22938 ts_tablet_manager.cc:1905] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:47.778494 22938 ts_tablet_manager.cc:1918] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:47.778601 22938 log.cc:1196] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/d0241f24edaa4764911809ce090babdc
I20260808 04:50:47.778904 22938 ts_tablet_manager.cc:1939] T d0241f24edaa4764911809ce090babdc P 015d29dab37440d9973491e1d6762a59: Deleting consensus metadata
I20260808 04:50:47.789323 22121 catalog_manager.cc:4841] TS 015d29dab37440d9973491e1d6762a59 (127.15.30.66:43975): tablet d0241f24edaa4764911809ce090babdc (table test-table [id=dc675d4c5a7d4bf8b0cd35d49fab5f44]) successfully deleted
I20260808 04:50:47.790984 22933 ts_tablet_manager.cc:1905] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:47.793205 22933 ts_tablet_manager.cc:1918] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:47.793327 22933 log.cc:1196] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/d0241f24edaa4764911809ce090babdc
I20260808 04:50:47.793663 22933 ts_tablet_manager.cc:1939] T d0241f24edaa4764911809ce090babdc P 0979066182104a0fa0246ddae9f327a4: Deleting consensus metadata
I20260808 04:50:47.794756 22121 catalog_manager.cc:4841] TS 0979066182104a0fa0246ddae9f327a4 (127.15.30.67:39389): tablet d0241f24edaa4764911809ce090babdc (table test-table [id=dc675d4c5a7d4bf8b0cd35d49fab5f44]) successfully deleted
W20260808 04:50:47.795244 22118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:41007 (error 108)
I20260808 04:50:47.796113 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 22466
I20260808 04:50:47.796267 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:47.795969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:41007 (local address 127.0.0.1:57472)
0808 04:50:47.796112 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:47.796184 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:41007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 04:50:47.798009 22118 catalog_manager.cc:4584] TS e112e7b61daf4e9ebf37cb33657e8bdd (127.15.30.65:41007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0241f24edaa4764911809ce090babdc: Network error: Client connection negotiation failed: client connection to 127.15.30.65:41007: connect: Connection refused (error 111)
I20260808 04:50:47.804667 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 22675
I20260808 04:50:47.820295 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 22093
2026-08-08T04:50:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3336 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T04:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:47Z Disabled control of system clock
I20260808 04:50:47.858664 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33771
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:50:47.969236 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:47.969471 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:47.969506 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:47.973333 22944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:47.973410 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:47.973443 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:47.973482 22944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:47.973519 22944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:47.978112 22944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:47.979066 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:47.980340 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:47.988456 22951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:47.988466 22952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:47.988507 22954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:47.988741 22944 server_base.cc:1034] running on GCE node
I20260808 04:50:47.989419 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:47.990399 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:47.991602 22944 hybrid_clock.cc:648] HybridClock initialized: now 1786164647991578 us; error 30 us; skew 500 ppm
I20260808 04:50:47.993414 22944 webserver.cc:460] Webserver started at http://127.15.30.126:41367/ using document root <none> and password file <none>
I20260808 04:50:47.994028 22944 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:47.994133 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:47.994364 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:47.996294 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "994d824ea76f054484b728ec8ae5007f"
server_key_iv: "edebbf83e2518acd257caebaa926f7fc"
server_key_version: "encryptionkey@0"
I20260808 04:50:47.996948 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "994d824ea76f054484b728ec8ae5007f"
server_key_iv: "edebbf83e2518acd257caebaa926f7fc"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.000432 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:50:48.002928 22962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.004259 22944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:48.004460 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c"
format_stamp: "Formatted at 2026-08-08 04:50:47 on dist-test-slave-zpvx"
server_key: "994d824ea76f054484b728ec8ae5007f"
server_key_iv: "edebbf83e2518acd257caebaa926f7fc"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.004581 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:48.008927 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:48.009495 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:48.009685 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:48.016583 22944 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33771
I20260808 04:50:48.016621 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33771 every 8 connection(s)
I20260808 04:50:48.017495 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:48.020539 23056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:48.024199 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 22944
I20260808 04:50:48.024340 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:48.024641 15481 external_mini_cluster.cc:1428] Setting key b367a8648d452f6eae9d02c6a0cf2a55
I20260808 04:50:48.025977 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c: Bootstrap starting.
I20260808 04:50:48.028726 23056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:48.029637 23056 log.cc:824] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:48.031131 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.026501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.15.30.126:33771)
0808 04:50:48.027220 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.027243 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.027286 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:50:48.028084 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.028107 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.028560 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.028939 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.029667 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.029673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.029687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.029710 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.029785 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.029809 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.029915 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.030032 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.030288 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":104,"threads_started":1}
I20260808 04:50:48.031772 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c: No bootstrap required, opened a new log
I20260808 04:50:48.035449 23056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33771 } }
I20260808 04:50:48.035640 23056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:48.035681 23056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2ba3066e344bd6b3ab1cff54bf0b2c, State: Initialized, Role: FOLLOWER
I20260808 04:50:48.036057 23056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33771 } }
I20260808 04:50:48.036172 23056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:48.036226 23056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:48.036294 23056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:48.037096 23056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33771 } }
I20260808 04:50:48.037422 23056 leader_election.cc:302] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a2ba3066e344bd6b3ab1cff54bf0b2c; no voters:
I20260808 04:50:48.037706 23056 leader_election.cc:288] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:48.037801 23063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:48.038069 23063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [term 1 LEADER]: Becoming Leader. State: Replica: 9a2ba3066e344bd6b3ab1cff54bf0b2c, State: Running, Role: LEADER
I20260808 04:50:48.038407 23063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33771 } }
I20260808 04:50:48.038944 23056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:48.040287 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a2ba3066e344bd6b3ab1cff54bf0b2c. Latest consensus state: current_term: 1 leader_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33771 } } }
I20260808 04:50:48.040270 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2ba3066e344bd6b3ab1cff54bf0b2c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33771 } } }
I20260808 04:50:48.040450 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:48.040450 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c [sys.catalog]: This master's current role is: LEADER
W20260808 04:50:48.043773 23089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:50:48.043905 23089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:50:48.044003 23081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:48.044834 23081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:48.050201 23081 catalog_manager.cc:1347] Generated new cluster ID: eeff5f88798d456ba56f073b983c4234
I20260808 04:50:48.050267 23081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:48.080785 23081 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:48.081143 23081 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:48.094642 23081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c: Generated new TSK 0
I20260808 04:50:48.095341 23081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:50:48.109618 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33771
--builtin_ntp_servers=127.15.30.84:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:50:48.224915 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:48.225162 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:48.225200 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:48.229748 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:48.229799 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:48.229888 23095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:48.234616 23095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:48.235670 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:48.237107 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:48.247211 23105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:48.247207 23111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:48.247409 23106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:48.247421 23095 server_base.cc:1034] running on GCE node
I20260808 04:50:48.248044 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:48.248744 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:48.249912 23095 hybrid_clock.cc:648] HybridClock initialized: now 1786164648249893 us; error 32 us; skew 500 ppm
I20260808 04:50:48.251855 23095 webserver.cc:460] Webserver started at http://127.15.30.65:39215/ using document root <none> and password file <none>
I20260808 04:50:48.252521 23095 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:48.252614 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:48.252861 23095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:48.255100 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "554bd25c80584282a3f1382384c75064"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "b51d997079a1f29a8810e3806971b150"
server_key_iv: "f341c0bc38500179201d9a88775f5deb"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.255692 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "554bd25c80584282a3f1382384c75064"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "b51d997079a1f29a8810e3806971b150"
server_key_iv: "f341c0bc38500179201d9a88775f5deb"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.259397 23095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:48.261996 23120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.263187 23095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:48.263322 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "554bd25c80584282a3f1382384c75064"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "b51d997079a1f29a8810e3806971b150"
server_key_iv: "f341c0bc38500179201d9a88775f5deb"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.263425 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:48.267925 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:48.268559 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:48.268744 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:48.269302 23095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:48.270185 23095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:48.270236 23095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.270298 23095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:48.270330 23095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.278774 23095 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:34009
I20260808 04:50:48.278807 23292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:34009 every 8 connection(s)
I20260808 04:50:48.279593 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:48.285324 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.281385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:42527 (local address 127.15.30.126:33771)
0808 04:50:48.281519 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.281523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.282428 (+ 905us) server_negotiation.cc:407] Connection header received
0808 04:50:48.283766 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.283769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.283806 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.283879 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.284963 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.284966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.284967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.284977 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.284992 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.284999 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.285068 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.285171 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.285206 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:50:48.286105 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.281661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33771 (local address 127.15.30.65:42527)
0808 04:50:48.282289 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.282327 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.283563 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.284038 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.284043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.284125 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.284732 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.284734 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.284772 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.284836 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.285063 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.285068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.285071 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.285190 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.285387 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260808 04:50:48.286677 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 23095
I20260808 04:50:48.286835 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:48.287213 15481 external_mini_cluster.cc:1428] Setting key 9f37b35a538bd8b0a23ac9aa435b9b7a
I20260808 04:50:48.287518 23293 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33771
I20260808 04:50:48.287746 23293 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:48.288169 23293 heartbeater.cc:511] Master 127.15.30.126:33771 requested a full tablet report, sending...
I20260808 04:50:48.289295 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33771
--builtin_ntp_servers=127.15.30.84:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 04:50:48.289819 22991 ts_manager.cc:194] Registered new tserver with Master: 554bd25c80584282a3f1382384c75064 (127.15.30.65:34009)
W20260808 04:50:48.398036 23297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:48.398257 23297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:48.398294 23297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:48.402275 23297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:48.402321 23297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:48.402411 23297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:48.407401 23297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:48.408433 23297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:48.409808 23297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:48.419183 23305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:48.419167 23311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:48.419167 23302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:48.419214 23297 server_base.cc:1034] running on GCE node
I20260808 04:50:48.419783 23297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:48.420416 23297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:48.421612 23297 hybrid_clock.cc:648] HybridClock initialized: now 1786164648421599 us; error 35 us; skew 500 ppm
I20260808 04:50:48.423326 23297 webserver.cc:460] Webserver started at http://127.15.30.66:34859/ using document root <none> and password file <none>
I20260808 04:50:48.423882 23297 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:48.423974 23297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:48.424218 23297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:48.426260 23297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "d2fc4b42feab447ab95077c53a0ef2f3"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "7c13ac49db535c4bbe1b179a3a29d00c"
server_key_iv: "477a512a3352138b9022c17c0826ac20"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.426803 23297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "d2fc4b42feab447ab95077c53a0ef2f3"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "7c13ac49db535c4bbe1b179a3a29d00c"
server_key_iv: "477a512a3352138b9022c17c0826ac20"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.430343 23297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:50:48.432745 23321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.433890 23297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:48.434029 23297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "d2fc4b42feab447ab95077c53a0ef2f3"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "7c13ac49db535c4bbe1b179a3a29d00c"
server_key_iv: "477a512a3352138b9022c17c0826ac20"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.434170 23297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:48.438460 23297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:48.439061 23297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:48.439234 23297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:48.439803 23297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:48.440779 23297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:48.440845 23297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.440908 23297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:48.440938 23297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.449194 23297 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:43475
I20260808 04:50:48.449272 23504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:43475 every 8 connection(s)
I20260808 04:50:48.450031 23297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:48.455888 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.451773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:38307 (local address 127.15.30.126:33771)
0808 04:50:48.451915 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.451919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.452851 (+ 932us) server_negotiation.cc:407] Connection header received
0808 04:50:48.454268 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.454271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.454311 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.454388 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.455402 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.455405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.455407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.455420 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.455439 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.455447 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.455525 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.455646 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.455738 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:50:48.455924 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 23297
I20260808 04:50:48.456028 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:48.456358 15481 external_mini_cluster.cc:1428] Setting key 56398663f179766194313db01003fa26
I20260808 04:50:48.456717 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.452065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33771 (local address 127.15.30.66:38307)
0808 04:50:48.452728 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.452766 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.454058 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.454539 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.454546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.454644 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.455150 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.455152 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.455190 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.455253 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.455525 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.455532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.455536 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.455673 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.455849 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260808 04:50:48.457930 23506 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33771
I20260808 04:50:48.458156 23506 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:48.458617 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33771
--builtin_ntp_servers=127.15.30.84:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 04:50:48.458630 23506 heartbeater.cc:511] Master 127.15.30.126:33771 requested a full tablet report, sending...
I20260808 04:50:48.459784 22991 ts_manager.cc:194] Registered new tserver with Master: d2fc4b42feab447ab95077c53a0ef2f3 (127.15.30.66:43475)
W20260808 04:50:48.571159 23511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:48.571398 23511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:48.571435 23511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:48.575441 23511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:48.575485 23511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:48.575562 23511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:48.580137 23511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:48.581166 23511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:48.582477 23511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:48.591789 23520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:48.591812 23511 server_base.cc:1034] running on GCE node
W20260808 04:50:48.591812 23522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:48.591789 23519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:48.592440 23511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:48.593034 23511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:48.594205 23511 hybrid_clock.cc:648] HybridClock initialized: now 1786164648594193 us; error 24 us; skew 500 ppm
I20260808 04:50:48.603006 23511 webserver.cc:460] Webserver started at http://127.15.30.67:34093/ using document root <none> and password file <none>
I20260808 04:50:48.603525 23511 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:48.603585 23511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:48.603736 23511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:48.605736 23511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "cbc794d4c62947a89c5b0c4840478f60"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "fed8b75e806d1bec271588ef6846c298"
server_key_iv: "d55ac7ffb64298508be21444cb706fb1"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.606489 23511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "cbc794d4c62947a89c5b0c4840478f60"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "fed8b75e806d1bec271588ef6846c298"
server_key_iv: "d55ac7ffb64298508be21444cb706fb1"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.610252 23511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:50:48.612701 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.613829 23511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:48.613947 23511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "cbc794d4c62947a89c5b0c4840478f60"
format_stamp: "Formatted at 2026-08-08 04:50:48 on dist-test-slave-zpvx"
server_key: "fed8b75e806d1bec271588ef6846c298"
server_key_iv: "d55ac7ffb64298508be21444cb706fb1"
server_key_version: "encryptionkey@0"
I20260808 04:50:48.614056 23511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:48.618590 23511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:48.619148 23511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:48.619330 23511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:48.619854 23511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:48.620791 23511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:48.620838 23511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.620903 23511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:48.620941 23511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:48.629619 23511 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:36611
I20260808 04:50:48.629655 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:36611 every 8 connection(s)
I20260808 04:50:48.630411 23511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:48.635896 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 23511
I20260808 04:50:48.635993 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.632092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:51363 (local address 127.15.30.126:33771)
0808 04:50:48.632276 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.632280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.633293 (+ 1013us) server_negotiation.cc:407] Connection header received
0808 04:50:48.634493 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.634495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.634531 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.634603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.635502 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.635505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.635507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.635518 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.635537 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.635544 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.635680 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.635795 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.635842 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:50:48.636014 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:48.636277 15481 external_mini_cluster.cc:1428] Setting key d4f29d74aa4731c60d3fa2c5426ce8b2
I20260808 04:50:48.636984 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.632519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33771 (local address 127.15.30.67:51363)
0808 04:50:48.633125 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.633187 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.634318 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.634721 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.634727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.634806 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.635264 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.635265 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.635301 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.635365 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.635659 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.635665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.635668 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.635793 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.635999 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260808 04:50:48.638135 23732 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33771
I20260808 04:50:48.638362 23732 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:48.638785 23732 heartbeater.cc:511] Master 127.15.30.126:33771 requested a full tablet report, sending...
I20260808 04:50:48.639873 22991 ts_manager.cc:194] Registered new tserver with Master: cbc794d4c62947a89c5b0c4840478f60 (127.15.30.67:36611)
I20260808 04:50:48.649977 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:50:48.653637 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.651933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.15.30.126:33771)
0808 04:50:48.652095 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.652099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.652304 (+ 205us) server_negotiation.cc:407] Connection header received
0808 04:50:48.652475 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.652478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.652519 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.652606 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.653269 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.653272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.653275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.653287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.653305 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.653320 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.653410 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.653483 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.653521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:50:48.656509 22991 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:48.659428 22991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:48.671013 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.668870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.15.30.67:36611)
0808 04:50:48.669340 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.669352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.669375 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:50:48.669423 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.669428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.669603 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.669807 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.670445 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.670450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.670458 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.670466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.670496 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.670511 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.670636 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.670813 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.670898 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":146,"threads_started":1}
I20260808 04:50:48.670984 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.667970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:36611 (local address 127.0.0.1:56852)
0808 04:50:48.668947 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.668964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.669066 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.669788 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.669790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.669805 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.670272 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.670273 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.670282 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.670303 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.670622 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.670634 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.670644 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.670763 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.670808 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":100,"threads_started":1}
I20260808 04:50:48.671382 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.668455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.15.30.66:43475)
0808 04:50:48.668849 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.668863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.668891 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:50:48.668950 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.668956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.669176 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.669339 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.670447 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.670455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.670467 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.670480 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.670522 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.670551 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.671088 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.671142 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.671212 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260808 04:50:48.670984 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.667714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:43475 (local address 127.0.0.1:48270)
0808 04:50:48.668502 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.668519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.668593 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.669363 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.669369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.669447 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.670032 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.670034 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.670054 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.670130 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.670653 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.670656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.670659 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.670737 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.670813 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":40,"thread_start_us":100,"threads_started":1}
I20260808 04:50:48.672171 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.667553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34009 (local address 127.0.0.1:50730)
0808 04:50:48.667916 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.667967 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.668068 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.670366 (+ 2298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.670370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.670385 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.671232 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.671234 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.671244 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.671269 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.671724 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.671726 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.671729 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.672034 (+ 305us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.672066 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260808 04:50:48.672679 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.668537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.15.30.65:34009)
0808 04:50:48.669248 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.669262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.669290 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:50:48.669342 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.669348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.669522 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.669659 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.671467 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.671473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.671484 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.671496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.671536 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.671573 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.671676 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.672462 (+ 786us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.672521 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":131,"threads_started":1}
I20260808 04:50:48.674376 23398 tablet_service.cc:1467] Processing CreateTablet for tablet 199a870dda8a479d8d6e8144075d9433 (DEFAULT_TABLE table=test-table [id=953b721427c84b32ad6fa0817abdc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:48.674356 23618 tablet_service.cc:1467] Processing CreateTablet for tablet 199a870dda8a479d8d6e8144075d9433 (DEFAULT_TABLE table=test-table [id=953b721427c84b32ad6fa0817abdc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:48.675473 23398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199a870dda8a479d8d6e8144075d9433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:48.675472 23618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199a870dda8a479d8d6e8144075d9433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:48.677073 23193 tablet_service.cc:1467] Processing CreateTablet for tablet 199a870dda8a479d8d6e8144075d9433 (DEFAULT_TABLE table=test-table [id=953b721427c84b32ad6fa0817abdc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:48.678027 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199a870dda8a479d8d6e8144075d9433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:48.681488 23760 tablet_bootstrap.cc:492] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Bootstrap starting.
I20260808 04:50:48.682945 23762 tablet_bootstrap.cc:492] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Bootstrap starting.
I20260808 04:50:48.683809 23760 tablet_bootstrap.cc:654] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:48.684751 23760 log.cc:824] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:48.684934 23763 tablet_bootstrap.cc:492] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: Bootstrap starting.
I20260808 04:50:48.687009 23760 tablet_bootstrap.cc:492] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: No bootstrap required, opened a new log
I20260808 04:50:48.687052 23763 tablet_bootstrap.cc:654] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:48.687211 23760 ts_tablet_manager.cc:1397] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:50:48.687853 23763 log.cc:824] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:48.690332 23763 tablet_bootstrap.cc:492] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: No bootstrap required, opened a new log
I20260808 04:50:48.690546 23763 ts_tablet_manager.cc:1397] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:50:48.691390 23762 tablet_bootstrap.cc:654] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:48.691742 23760 raft_consensus.cc:348] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.691989 23760 raft_consensus.cc:374] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:48.692034 23760 raft_consensus.cc:729] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc794d4c62947a89c5b0c4840478f60, State: Initialized, Role: FOLLOWER
I20260808 04:50:48.692227 23762 log.cc:824] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:48.692541 23760 consensus_queue.cc:260] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.693493 23732 heartbeater.cc:503] Master 127.15.30.126:33771 was elected leader, sending a full tablet report...
I20260808 04:50:48.693583 23760 ts_tablet_manager.cc:1428] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:50:48.694926 23763 raft_consensus.cc:348] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.695173 23763 raft_consensus.cc:374] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:48.695235 23763 raft_consensus.cc:729] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554bd25c80584282a3f1382384c75064, State: Initialized, Role: FOLLOWER
I20260808 04:50:48.695353 23762 tablet_bootstrap.cc:492] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: No bootstrap required, opened a new log
I20260808 04:50:48.695556 23762 ts_tablet_manager.cc:1397] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260808 04:50:48.695668 23763 consensus_queue.cc:260] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.696347 23293 heartbeater.cc:503] Master 127.15.30.126:33771 was elected leader, sending a full tablet report...
I20260808 04:50:48.696919 23763 ts_tablet_manager.cc:1428] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:50:48.699819 23762 raft_consensus.cc:348] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.700065 23762 raft_consensus.cc:374] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:48.700135 23762 raft_consensus.cc:729] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fc4b42feab447ab95077c53a0ef2f3, State: Initialized, Role: FOLLOWER
I20260808 04:50:48.700578 23762 consensus_queue.cc:260] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.701195 23506 heartbeater.cc:503] Master 127.15.30.126:33771 was elected leader, sending a full tablet report...
I20260808 04:50:48.701408 23762 ts_tablet_manager.cc:1428] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:50:48.768798 23769 raft_consensus.cc:482] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:48.768987 23769 raft_consensus.cc:504] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.770044 23769 leader_election.cc:288] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554bd25c80584282a3f1382384c75064 (127.15.30.65:34009), cbc794d4c62947a89c5b0c4840478f60 (127.15.30.67:36611)
I20260808 04:50:48.771981 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.770315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:36611 (local address 127.15.30.66:44445)
0808 04:50:48.770417 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.770433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.770513 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.770804 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.770807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.770823 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.771366 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.771368 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.771381 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.771415 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.771712 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.771715 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.771719 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.771811 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.771851 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 04:50:48.772028 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.770423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:44445 (local address 127.15.30.67:36611)
0808 04:50:48.770547 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.770552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.770571 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:50:48.770622 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.770628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.770670 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.770766 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.771557 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.771563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.771569 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.771585 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.771609 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.771621 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.771716 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.771830 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.771875 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:50:48.772322 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.770315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34009 (local address 127.15.30.66:46933)
0808 04:50:48.770875 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.770891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:48.770997 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:48.771260 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:48.771262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:48.771274 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:48.771716 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:50:48.771718 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:48.771729 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:50:48.771755 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:48.772012 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:48.772017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:48.772024 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:50:48.772147 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:50:48.772184 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":164,"thread_start_us":135,"threads_started":1}
I20260808 04:50:48.772320 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.770398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:46933 (local address 127.15.30.65:34009)
0808 04:50:48.770519 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.770524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.771015 (+ 491us) server_negotiation.cc:407] Connection header received
0808 04:50:48.771085 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.771089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.771126 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.771204 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.771875 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.771878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.771881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.771892 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.771909 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.771918 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.772005 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.772168 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.772207 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 04:50:48.772691 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a870dda8a479d8d6e8144075d9433" candidate_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc794d4c62947a89c5b0c4840478f60" is_pre_election: true
I20260808 04:50:48.773032 23649 raft_consensus.cc:2454] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2fc4b42feab447ab95077c53a0ef2f3 in term 0.
I20260808 04:50:48.773062 23225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a870dda8a479d8d6e8144075d9433" candidate_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554bd25c80584282a3f1382384c75064" is_pre_election: true
I20260808 04:50:48.773325 23225 raft_consensus.cc:2454] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2fc4b42feab447ab95077c53a0ef2f3 in term 0.
I20260808 04:50:48.773520 23324 leader_election.cc:302] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbc794d4c62947a89c5b0c4840478f60, d2fc4b42feab447ab95077c53a0ef2f3; no voters:
I20260808 04:50:48.773800 23769 raft_consensus.cc:2781] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:48.773887 23769 raft_consensus.cc:482] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:48.773931 23769 raft_consensus.cc:3037] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:48.774967 23769 raft_consensus.cc:504] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
I20260808 04:50:48.775274 23769 leader_election.cc:288] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [CANDIDATE]: Term 1 election: Requested vote from peers 554bd25c80584282a3f1382384c75064 (127.15.30.65:34009), cbc794d4c62947a89c5b0c4840478f60 (127.15.30.67:36611)
I20260808 04:50:48.775593 23225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a870dda8a479d8d6e8144075d9433" candidate_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554bd25c80584282a3f1382384c75064"
I20260808 04:50:48.775586 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199a870dda8a479d8d6e8144075d9433" candidate_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc794d4c62947a89c5b0c4840478f60"
I20260808 04:50:48.775705 23225 raft_consensus.cc:3037] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:48.775733 23649 raft_consensus.cc:3037] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:48.777025 23225 raft_consensus.cc:2454] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2fc4b42feab447ab95077c53a0ef2f3 in term 1.
I20260808 04:50:48.777025 23649 raft_consensus.cc:2454] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2fc4b42feab447ab95077c53a0ef2f3 in term 1.
I20260808 04:50:48.777343 23326 leader_election.cc:302] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554bd25c80584282a3f1382384c75064, d2fc4b42feab447ab95077c53a0ef2f3; no voters:
I20260808 04:50:48.777527 23769 raft_consensus.cc:2781] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:48.777725 23769 raft_consensus.cc:686] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 1 LEADER]: Becoming Leader. State: Replica: d2fc4b42feab447ab95077c53a0ef2f3, State: Running, Role: LEADER
I20260808 04:50:48.778029 23769 consensus_queue.cc:237] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } }
W20260808 04:50:48.780755 23294 tablet.cc:2369] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:48.781083 22991 catalog_manager.cc:5495] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2fc4b42feab447ab95077c53a0ef2f3 (127.15.30.66). New cstate: current_term: 1 leader_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2fc4b42feab447ab95077c53a0ef2f3" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 } health_report { overall_health: UNKNOWN } } }
I20260808 04:50:48.791488 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.790097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.15.30.66:43475)
0808 04:50:48.790210 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:48.790213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:48.790255 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:50:48.790368 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:48.790370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:48.790422 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:48.790485 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:48.791056 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:48.791060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:48.791062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:48.791072 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:48.791087 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:48.791095 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:48.791164 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:48.791275 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 04:50:48.791336 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:50:48.796902 23225 raft_consensus.cc:1261] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 1 FOLLOWER]: Refusing update from remote peer d2fc4b42feab447ab95077c53a0ef2f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:48.796902 23649 raft_consensus.cc:1261] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 1 FOLLOWER]: Refusing update from remote peer d2fc4b42feab447ab95077c53a0ef2f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:48.797552 23773 consensus_queue.cc:1035] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc794d4c62947a89c5b0c4840478f60" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:48.797755 23769 consensus_queue.cc:1035] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554bd25c80584282a3f1382384c75064" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:48.804689 23777 mvcc.cc:205] Tried to move back new op lower bound from 7316130401465528320 to 7316130401397440512. Current Snapshot: MvccSnapshot[applied={T|T < 7316130401465528320}]
I20260808 04:50:48.805395 23778 mvcc.cc:205] Tried to move back new op lower bound from 7316130401465528320 to 7316130401397440512. Current Snapshot: MvccSnapshot[applied={T|T < 7316130401465528320}]
I20260808 04:50:48.805979 23780 mvcc.cc:205] Tried to move back new op lower bound from 7316130401465528320 to 7316130401397440512. Current Snapshot: MvccSnapshot[applied={T|T < 7316130401465528320}]
I20260808 04:50:48.813197 22990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:48.813360 22990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:48.815589 22990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a2ba3066e344bd6b3ab1cff54bf0b2c: Sending DeleteTablet for 3 replicas of tablet 199a870dda8a479d8d6e8144075d9433
I20260808 04:50:48.816334 23193 tablet_service.cc:1514] Processing DeleteTablet for tablet 199a870dda8a479d8d6e8144075d9433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:48 UTC) from {username='slave'} at 127.0.0.1:50730
I20260808 04:50:48.816397 23398 tablet_service.cc:1514] Processing DeleteTablet for tablet 199a870dda8a479d8d6e8144075d9433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:48 UTC) from {username='slave'} at 127.0.0.1:48270
I20260808 04:50:48.816397 23618 tablet_service.cc:1514] Processing DeleteTablet for tablet 199a870dda8a479d8d6e8144075d9433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:48 UTC) from {username='slave'} at 127.0.0.1:56852
I20260808 04:50:48.816908 23792 tablet_replica.cc:330] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064: stopping tablet replica
I20260808 04:50:48.817015 23795 tablet_replica.cc:330] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: stopping tablet replica
I20260808 04:50:48.817312 23792 raft_consensus.cc:2229] T 199a870dda8a479d8d6e8144075d9433 P 554bd25c80584282a3f1382384c75064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:48.817368 23795 raft_consensus.cc:2229] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:48.817637 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 23095
I20260808 04:50:48.817674 23795 raft_consensus.cc:2258] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:48.820859 23793 tablet_replica.cc:330] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: stopping tablet replica
I20260808 04:50:48.821197 23793 raft_consensus.cc:2229] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:48.821615 23793 raft_consensus.cc:2258] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:48.825179 23793 ts_tablet_manager.cc:1905] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:48.825178 23795 ts_tablet_manager.cc:1905] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:48.827971 23793 ts_tablet_manager.cc:1918] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:48.828071 23793 log.cc:1196] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/199a870dda8a479d8d6e8144075d9433
I20260808 04:50:48.828434 23793 ts_tablet_manager.cc:1939] T 199a870dda8a479d8d6e8144075d9433 P d2fc4b42feab447ab95077c53a0ef2f3: Deleting consensus metadata
I20260808 04:50:48.828544 23795 ts_tablet_manager.cc:1918] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:48.828648 23795 log.cc:1196] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/199a870dda8a479d8d6e8144075d9433
I20260808 04:50:48.828982 23795 ts_tablet_manager.cc:1939] T 199a870dda8a479d8d6e8144075d9433 P cbc794d4c62947a89c5b0c4840478f60: Deleting consensus metadata
I20260808 04:50:48.829579 22964 catalog_manager.cc:4841] TS d2fc4b42feab447ab95077c53a0ef2f3 (127.15.30.66:43475): tablet 199a870dda8a479d8d6e8144075d9433 (table test-table [id=953b721427c84b32ad6fa0817abdc7b8]) successfully deleted
I20260808 04:50:48.830138 22963 catalog_manager.cc:4841] TS cbc794d4c62947a89c5b0c4840478f60 (127.15.30.67:36611): tablet 199a870dda8a479d8d6e8144075d9433 (table test-table [id=953b721427c84b32ad6fa0817abdc7b8]) successfully deleted
W20260808 04:50:48.832178 22965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:34009 (error 108)
I20260808 04:50:48.833055 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 23297
I20260808 04:50:48.833259 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:48.832939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34009 (local address 127.0.0.1:50740)
0808 04:50:48.833062 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:48.833148 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 04:50:48.845400 22965 catalog_manager.cc:4584] TS 554bd25c80584282a3f1382384c75064 (127.15.30.65:34009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 199a870dda8a479d8d6e8144075d9433: Network error: Client connection negotiation failed: client connection to 127.15.30.65:34009: connect: Connection refused (error 111)
I20260808 04:50:48.851423 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 23511
I20260808 04:50:48.861856 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 22944
2026-08-08T04:50:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1038 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 04:50:48.889140 15481 mini_oidc.cc:152] Starting JWKS server
I20260808 04:50:48.890851 15481 webserver.cc:460] Webserver started at https://127.0.0.1:43215/ using document root <none> and password file <none>
I20260808 04:50:48.891044 15481 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:50:48.891324 15481 webserver.cc:460] Webserver started at http://127.0.0.1:42845/ using document root <none> and password file <none>
2026-08-08T04:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:48Z Disabled control of system clock
I20260808 04:50:48.901216 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:45445
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:45445
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/testchainca.cert with env {}
W20260808 04:50:49.009899 23805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:49.010128 23805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:49.010164 23805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:49.013931 23805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:49.013972 23805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:49.013988 23805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:49.014002 23805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:49.014017 23805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:50:49.014032 23805 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:50:49.014048 23805 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43215/jwks.json
I20260808 04:50:49.019085 23805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:45445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:45445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:49.019939 23805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:49.020828 23805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:49.028939 23818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:49.028947 23815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:49.028939 23814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:49.029223 23805 server_base.cc:1034] running on GCE node
I20260808 04:50:49.029881 23805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:49.030893 23805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:49.032078 23805 hybrid_clock.cc:648] HybridClock initialized: now 1786164649032066 us; error 24 us; skew 500 ppm
I20260808 04:50:49.033916 23805 webserver.cc:460] Webserver started at http://127.15.30.126:36559/ using document root <none> and password file <none>
I20260808 04:50:49.034473 23805 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:49.034566 23805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:49.034785 23805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:49.036815 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "d78faed7014a4bfbac534455bce3bdbe"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "1e8440791cad840691b1fd5fd5f96e50"
server_key_iv: "9a4b49dbb52f9e7f54934999549ee357"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.037374 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "d78faed7014a4bfbac534455bce3bdbe"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "1e8440791cad840691b1fd5fd5f96e50"
server_key_iv: "9a4b49dbb52f9e7f54934999549ee357"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.040831 23805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:50:49.043082 23826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:49.044229 23805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:49.044430 23805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "d78faed7014a4bfbac534455bce3bdbe"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "1e8440791cad840691b1fd5fd5f96e50"
server_key_iv: "9a4b49dbb52f9e7f54934999549ee357"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.044540 23805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:49.071799 23805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:49.072525 23805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:49.072713 23805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:49.079728 23805 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:45445
I20260808 04:50:49.079746 23914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:45445 every 8 connection(s)
I20260808 04:50:49.080751 23805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:49.083951 23915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:49.086851 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 23805
I20260808 04:50:49.086995 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:49.087270 15481 external_mini_cluster.cc:1428] Setting key 34ae6a533687ae2cbb9bd775ffd3447a
I20260808 04:50:49.089834 23915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe: Bootstrap starting.
I20260808 04:50:49.092101 23915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:49.092897 23915 log.cc:824] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:49.094749 23915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe: No bootstrap required, opened a new log
I20260808 04:50:49.096416 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:49.088594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.15.30.126:45445)
0808 04:50:49.089235 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:49.089251 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:49.089284 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:50:49.089896 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:49.089912 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:49.090336 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:49.090657 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:49.091082 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.092032 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.093033 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.093685 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.093868 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.094714 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:49.094720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:49.094747 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:49.095169 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:49.095218 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:49.095239 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:49.095350 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:49.095472 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:50:49.095719 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":146,"threads_started":1}
I20260808 04:50:49.098790 23915 raft_consensus.cc:348] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78faed7014a4bfbac534455bce3bdbe" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 45445 } }
I20260808 04:50:49.099042 23915 raft_consensus.cc:374] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:49.099107 23915 raft_consensus.cc:729] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78faed7014a4bfbac534455bce3bdbe, State: Initialized, Role: FOLLOWER
I20260808 04:50:49.099179 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:50:49.099586 23915 consensus_queue.cc:260] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78faed7014a4bfbac534455bce3bdbe" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 45445 } }
I20260808 04:50:49.099713 23915 raft_consensus.cc:388] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:49.099803 23915 raft_consensus.cc:482] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:49.099931 23915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:49.100824 23915 raft_consensus.cc:504] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78faed7014a4bfbac534455bce3bdbe" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 45445 } }
I20260808 04:50:49.101157 23915 leader_election.cc:302] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d78faed7014a4bfbac534455bce3bdbe; no voters:
I20260808 04:50:49.101404 23915 leader_election.cc:288] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:49.101544 23923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:49.101787 23923 raft_consensus.cc:686] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [term 1 LEADER]: Becoming Leader. State: Replica: d78faed7014a4bfbac534455bce3bdbe, State: Running, Role: LEADER
I20260808 04:50:49.102092 23923 consensus_queue.cc:237] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78faed7014a4bfbac534455bce3bdbe" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 45445 } }
I20260808 04:50:49.102550 23915 sys_catalog.cc:564] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:49.103660 23925 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d78faed7014a4bfbac534455bce3bdbe. Latest consensus state: current_term: 1 leader_uuid: "d78faed7014a4bfbac534455bce3bdbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78faed7014a4bfbac534455bce3bdbe" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 45445 } } }
I20260808 04:50:49.103804 23925 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:49.104066 23924 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d78faed7014a4bfbac534455bce3bdbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78faed7014a4bfbac534455bce3bdbe" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 45445 } } }
I20260808 04:50:49.104193 23924 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:49.104158 23935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:49.107391 23935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:49.113001 23935 catalog_manager.cc:1347] Generated new cluster ID: ab04f43558954ffa9813fef429d0d941
I20260808 04:50:49.113065 23935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:49.122650 23935 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:49.123453 23935 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:49.134546 23935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d78faed7014a4bfbac534455bce3bdbe: Generated new TSK 0
I20260808 04:50:49.135195 23935 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:50:49.174492 23959 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 04:50:49.174652 23959 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260808 04:50:49.176265 23957 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:49.150630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:45445 (local address 127.0.0.1:43592)
0808 04:50:49.150994 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:49.151010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:49.151149 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:49.151487 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:49.151490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:49.151499 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:50:49.151626 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:49.151633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.152407 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.152413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:49.153381 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:49.153391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.153818 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.153822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:49.153918 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.153969 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:50:49.174913 (+ 20944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:50:49.175103 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.30.126:45445: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
I20260808 04:50:49.176820 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:49.150739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.15.30.126:45445)
0808 04:50:49.150887 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:49.150892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:49.151089 (+ 197us) server_negotiation.cc:407] Connection header received
0808 04:50:49.151305 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:49.151307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:49.151353 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:49.151436 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:50:49.151739 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.152269 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.153495 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.153706 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.153785 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.154146 (+ 361us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:50:49.174654 (+ 20508us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:50:49.176687 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 04:50:49.176930 23920 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43592: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.30.126:45445: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 04:50:49.177398 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 23805
2026-08-08T04:50:49Z chronyd exiting
I20260808 04:50:49.222783 15481 test_util.cc:175] -----------------------------------------------
I20260808 04:50:49.222875 15481 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786164620544947-15481-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 04:50:49.224022 15481 mini_oidc.cc:152] Starting JWKS server
I20260808 04:50:49.224984 15481 webserver.cc:460] Webserver started at https://127.0.0.1:34141/ using document root <none> and password file <none>
I20260808 04:50:49.225286 15481 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:50:49.225620 15481 webserver.cc:460] Webserver started at http://127.0.0.1:34777/ using document root <none> and password file <none>
2026-08-08T04:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:49Z Disabled control of system clock
I20260808 04:50:49.236140 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:38109
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:38109
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/testchainca.cert with env {}
W20260808 04:50:49.347018 23968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:49.347273 23968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:49.347313 23968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:49.351336 23968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:49.351392 23968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:49.351406 23968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:49.351423 23968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:49.351437 23968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:50:49.351490 23968 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:50:49.351511 23968 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34141/jwks.json
I20260808 04:50:49.356947 23968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:38109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:38109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:49.357944 23968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:49.358916 23968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:49.367427 23978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:49.367421 23977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:49.367478 23968 server_base.cc:1034] running on GCE node
W20260808 04:50:49.367426 23982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:49.368115 23968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:49.369242 23968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:49.370440 23968 hybrid_clock.cc:648] HybridClock initialized: now 1786164649370421 us; error 39 us; skew 500 ppm
I20260808 04:50:49.372218 23968 webserver.cc:460] Webserver started at http://127.15.30.126:33041/ using document root <none> and password file <none>
I20260808 04:50:49.372843 23968 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:49.372941 23968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:49.373167 23968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:49.375171 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "ce63052f085f49e9adb059afdabeb04f"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "2fb5dbf0e55f5c37c2492184bf35849f"
server_key_iv: "1f8352b8d161459d79560deb0aa6d189"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.375749 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "ce63052f085f49e9adb059afdabeb04f"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "2fb5dbf0e55f5c37c2492184bf35849f"
server_key_iv: "1f8352b8d161459d79560deb0aa6d189"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.379354 23968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:49.381875 23991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:49.383301 23968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:50:49.383467 23968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "ce63052f085f49e9adb059afdabeb04f"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "2fb5dbf0e55f5c37c2492184bf35849f"
server_key_iv: "1f8352b8d161459d79560deb0aa6d189"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.383626 23968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:49.428112 23968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:49.428961 23968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:49.429225 23968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:49.436664 23968 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:38109
I20260808 04:50:49.436698 24081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:38109 every 8 connection(s)
I20260808 04:50:49.437613 23968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:49.440932 24084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:49.442590 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 23968
I20260808 04:50:49.442716 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:49.443043 15481 external_mini_cluster.cc:1428] Setting key 059ff1dacf75761de8630bae951faeb5
I20260808 04:50:49.447711 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f: Bootstrap starting.
I20260808 04:50:49.450183 24084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:49.451002 24084 log.cc:824] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:49.452521 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:49.444652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.15.30.126:38109)
0808 04:50:49.445305 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:49.445321 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:49.445355 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:50:49.445996 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:49.446023 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:49.446419 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:49.446763 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:49.447181 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.448197 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.449149 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.449615 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.449790 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.450711 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:49.450716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:49.450728 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:49.451136 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:49.451209 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:49.451230 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:49.451340 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:49.451489 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 04:50:49.451752 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260808 04:50:49.453213 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f: No bootstrap required, opened a new log
I20260808 04:50:49.455255 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:50:49.457546 24084 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce63052f085f49e9adb059afdabeb04f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38109 } }
I20260808 04:50:49.457772 24084 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:49.457808 24084 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce63052f085f49e9adb059afdabeb04f, State: Initialized, Role: FOLLOWER
I20260808 04:50:49.458184 24084 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce63052f085f49e9adb059afdabeb04f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38109 } }
I20260808 04:50:49.458333 24084 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:49.458407 24084 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:49.458520 24084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:49.459599 24084 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce63052f085f49e9adb059afdabeb04f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38109 } }
I20260808 04:50:49.459959 24084 leader_election.cc:302] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce63052f085f49e9adb059afdabeb04f; no voters:
I20260808 04:50:49.460316 24084 leader_election.cc:288] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:49.460428 24091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:49.460728 24091 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [term 1 LEADER]: Becoming Leader. State: Replica: ce63052f085f49e9adb059afdabeb04f, State: Running, Role: LEADER
I20260808 04:50:49.461027 24091 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce63052f085f49e9adb059afdabeb04f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38109 } }
I20260808 04:50:49.461642 24084 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:49.462771 24092 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce63052f085f49e9adb059afdabeb04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce63052f085f49e9adb059afdabeb04f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38109 } } }
I20260808 04:50:49.462904 24092 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:49.463199 24093 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce63052f085f49e9adb059afdabeb04f. Latest consensus state: current_term: 1 leader_uuid: "ce63052f085f49e9adb059afdabeb04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce63052f085f49e9adb059afdabeb04f" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38109 } } }
I20260808 04:50:49.463305 24093 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:49.463388 24103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:49.466548 24103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:49.472472 24103 catalog_manager.cc:1347] Generated new cluster ID: 8dd7094cf3dc41afaaf5a8fa5977123e
I20260808 04:50:49.472543 24103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:49.497035 24103 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:49.497963 24103 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:49.502143 24103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce63052f085f49e9adb059afdabeb04f: Generated new TSK 0
I20260808 04:50:49.502815 24103 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:50:49.530222 24129 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 04:50:49.530318 24129 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:50:49.530943 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:49.515019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.15.30.126:38109)
0808 04:50:49.515168 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:49.515173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:49.515357 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:50:49.515520 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:49.515522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:49.515569 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:49.515659 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:50:49.515937 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.516551 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.517898 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.518109 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.518209 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.518522 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:50:49.530617 (+ 12095us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:50:49.530837 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 04:50:49.531025 24089 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 04:50:49.531175 24128 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:49.514927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:38109 (local address 127.0.0.1:34920)
0808 04:50:49.515256 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:49.515273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:49.515402 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:49.515692 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:49.515695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:49.515705 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:50:49.515825 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:49.515832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.516668 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.516673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:49.517776 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:49.517786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.518228 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.518232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:49.518332 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.518362 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:50:49.530804 (+ 12442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:50:49.530994 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.30.126:38109: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260808 04:50:49.532059 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 23968
2026-08-08T04:50:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (356 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 04:50:49.580014 15481 mini_oidc.cc:152] Starting JWKS server
I20260808 04:50:49.580951 15481 webserver.cc:460] Webserver started at https://127.0.0.1:41583/ using document root <none> and password file <none>
I20260808 04:50:49.581064 15481 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:50:49.581349 15481 webserver.cc:460] Webserver started at http://127.0.0.1:34503/ using document root <none> and password file <none>
2026-08-08T04:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:49Z Disabled control of system clock
I20260808 04:50:49.591395 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35229
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:35229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41583/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:50:49.702428 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:49.702667 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:49.702703 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:49.706677 24136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:49.706724 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:49.706740 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:49.706756 24136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:49.706770 24136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:50:49.706809 24136 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:50:49.706838 24136 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41583/jwks.json
I20260808 04:50:49.711392 24136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:35229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41583/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:49.712394 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:49.714148 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:49.722968 24146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:49.722961 24151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:49.722942 24148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:49.723001 24136 server_base.cc:1034] running on GCE node
I20260808 04:50:49.723806 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:49.724903 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:49.726107 24136 hybrid_clock.cc:648] HybridClock initialized: now 1786164649726094 us; error 29 us; skew 500 ppm
I20260808 04:50:49.727950 24136 webserver.cc:460] Webserver started at http://127.15.30.126:45371/ using document root <none> and password file <none>
I20260808 04:50:49.728592 24136 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:49.728688 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:49.728921 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:49.730896 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "f91269b8646f4bdabb5a39daf4e826d1"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "84ba0ba0f1395c65a87b0cc4d57fe00d"
server_key_iv: "e605f4e6a104c7195504ba82db2d21df"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.731458 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "f91269b8646f4bdabb5a39daf4e826d1"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "84ba0ba0f1395c65a87b0cc4d57fe00d"
server_key_iv: "e605f4e6a104c7195504ba82db2d21df"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.735062 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 04:50:49.737571 24162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:49.738842 24136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:49.739010 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "f91269b8646f4bdabb5a39daf4e826d1"
format_stamp: "Formatted at 2026-08-08 04:50:49 on dist-test-slave-zpvx"
server_key: "84ba0ba0f1395c65a87b0cc4d57fe00d"
server_key_iv: "e605f4e6a104c7195504ba82db2d21df"
server_key_version: "encryptionkey@0"
I20260808 04:50:49.739149 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:49.764978 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:49.765728 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:49.765936 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:49.772902 24136 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:35229
I20260808 04:50:49.772939 24247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:35229 every 8 connection(s)
I20260808 04:50:49.773882 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:49.777068 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:49.777165 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 24136
I20260808 04:50:49.777279 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:49.777611 15481 external_mini_cluster.cc:1428] Setting key ae90218adb13764f825126eeff55ca27
I20260808 04:50:49.783771 24258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1: Bootstrap starting.
I20260808 04:50:49.786027 24258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:49.786806 24258 log.cc:824] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:49.787020 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:49.778965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.15.30.126:35229)
0808 04:50:49.779670 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:49.779687 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:49.779724 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:50:49.780487 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:49.780506 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:49.780920 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:49.781277 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:49.781745 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.782714 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.783665 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.784176 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.784356 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.785287 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:49.785292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:49.785304 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:49.785685 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:49.785739 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:49.785759 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:49.785870 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:49.786007 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 04:50:49.786270 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":153,"threads_started":1}
I20260808 04:50:49.788950 24258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1: No bootstrap required, opened a new log
I20260808 04:50:49.789996 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:50:49.792832 24258 raft_consensus.cc:348] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91269b8646f4bdabb5a39daf4e826d1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35229 } }
I20260808 04:50:49.793056 24258 raft_consensus.cc:374] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:49.793135 24258 raft_consensus.cc:729] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91269b8646f4bdabb5a39daf4e826d1, State: Initialized, Role: FOLLOWER
I20260808 04:50:49.793536 24258 consensus_queue.cc:260] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91269b8646f4bdabb5a39daf4e826d1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35229 } }
I20260808 04:50:49.793689 24258 raft_consensus.cc:388] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:49.793781 24258 raft_consensus.cc:482] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:49.793900 24258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:49.794847 24258 raft_consensus.cc:504] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91269b8646f4bdabb5a39daf4e826d1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35229 } }
I20260808 04:50:49.795236 24258 leader_election.cc:302] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f91269b8646f4bdabb5a39daf4e826d1; no voters:
I20260808 04:50:49.795620 24258 leader_election.cc:288] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:49.795753 24270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:49.796001 24270 raft_consensus.cc:686] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [term 1 LEADER]: Becoming Leader. State: Replica: f91269b8646f4bdabb5a39daf4e826d1, State: Running, Role: LEADER
I20260808 04:50:49.796495 24270 consensus_queue.cc:237] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91269b8646f4bdabb5a39daf4e826d1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35229 } }
I20260808 04:50:49.796945 24258 sys_catalog.cc:564] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:49.798333 24271 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f91269b8646f4bdabb5a39daf4e826d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91269b8646f4bdabb5a39daf4e826d1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35229 } } }
I20260808 04:50:49.798346 24273 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f91269b8646f4bdabb5a39daf4e826d1. Latest consensus state: current_term: 1 leader_uuid: "f91269b8646f4bdabb5a39daf4e826d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91269b8646f4bdabb5a39daf4e826d1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35229 } } }
I20260808 04:50:49.798482 24271 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:49.798481 24273 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1 [sys.catalog]: This master's current role is: LEADER
W20260808 04:50:49.802100 24298 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:50:49.802170 24298 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:50:49.802250 24292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:49.802949 24292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:49.808616 24292 catalog_manager.cc:1347] Generated new cluster ID: 77219f341cea4ad8ac0fa3f812901df5
I20260808 04:50:49.808684 24292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:49.824390 24292 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:49.825258 24292 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:49.828769 24292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f91269b8646f4bdabb5a39daf4e826d1: Generated new TSK 0
I20260808 04:50:49.829399 24292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:50:49.845062 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:49.840436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.15.30.126:35229)
0808 04:50:49.840562 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:49.840567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:49.840751 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:50:49.840941 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:49.840943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:49.840989 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:49.841093 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:50:49.841352 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.841884 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.842790 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.843011 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.843126 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.843470 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:50:49.844714 (+ 1244us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:50:49.844966 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 04:50:49.845139 24267 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 04:50:49.848009 24315 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:49.840348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35229 (local address 127.0.0.1:57432)
0808 04:50:49.840663 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:49.840675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:49.840800 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:49.841115 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:49.841119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:49.841127 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:50:49.841241 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:49.841247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.842004 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.842008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:49.842660 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:49.842668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:49.843148 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:49.843152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:49.843245 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:49.843271 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:50:49.844870 (+ 1599us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:50:49.847802 (+ 2932us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.30.126:35229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":2740,"thread_start_us":100,"threads_started":1}
I20260808 04:50:49.848762 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 24136
E20260808 04:50:49.859527 24316 webserver.cc:533] Webserver: sslize failed:
E20260808 04:50:49.859591 24316 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T04:50:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T04:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:49Z Disabled control of system clock
I20260808 04:50:49.901403 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35367
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:35367
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/testchainca.cert with env {}
W20260808 04:50:50.009747 24321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:50.009986 24321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:50.010025 24321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:50.013976 24321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:50.014025 24321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:50.014041 24321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:50.014057 24321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:50.014071 24321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:50:50.014122 24321 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:50:50.014151 24321 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 04:50:50.019809 24321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:35367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:50.020779 24321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:50.021746 24321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:50.030155 24329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:50.030190 24333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:50.030275 24327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:50.030298 24321 server_base.cc:1034] running on GCE node
I20260808 04:50:50.030884 24321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:50.031893 24321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:50.033073 24321 hybrid_clock.cc:648] HybridClock initialized: now 1786164650033044 us; error 45 us; skew 500 ppm
I20260808 04:50:50.034780 24321 webserver.cc:460] Webserver started at http://127.15.30.126:45441/ using document root <none> and password file <none>
I20260808 04:50:50.035341 24321 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:50.035434 24321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:50.035663 24321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:50.037670 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "a768544b405a44c5a7c4b9b6fb07b658"
format_stamp: "Formatted at 2026-08-08 04:50:50 on dist-test-slave-zpvx"
server_key: "aecccde1d6971a3a5c6c1673fde40696"
server_key_iv: "748c9e15400e44017e7c893898313c70"
server_key_version: "encryptionkey@0"
I20260808 04:50:50.038250 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "a768544b405a44c5a7c4b9b6fb07b658"
format_stamp: "Formatted at 2026-08-08 04:50:50 on dist-test-slave-zpvx"
server_key: "aecccde1d6971a3a5c6c1673fde40696"
server_key_iv: "748c9e15400e44017e7c893898313c70"
server_key_version: "encryptionkey@0"
I20260808 04:50:50.041785 24321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:50:50.044121 24343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:50.045331 24321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:50.045475 24321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "a768544b405a44c5a7c4b9b6fb07b658"
format_stamp: "Formatted at 2026-08-08 04:50:50 on dist-test-slave-zpvx"
server_key: "aecccde1d6971a3a5c6c1673fde40696"
server_key_iv: "748c9e15400e44017e7c893898313c70"
server_key_version: "encryptionkey@0"
I20260808 04:50:50.045585 24321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:50.076857 24321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:50.077574 24321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:50.077790 24321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:50.084973 24321 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:35367
I20260808 04:50:50.084993 24437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:35367 every 8 connection(s)
I20260808 04:50:50.085923 24321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:50.087018 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 24321
I20260808 04:50:50.087153 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:50.087464 15481 external_mini_cluster.cc:1428] Setting key 84e6e7cbfcbd301076463c59d7ce2cbc
I20260808 04:50:50.089123 24438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:50.094664 24438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658: Bootstrap starting.
I20260808 04:50:50.096576 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:50.088883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.15.30.126:35367)
0808 04:50:50.089553 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:50.089569 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:50.089602 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:50:50.090217 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:50.090232 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:50.090592 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:50.090926 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:50.091341 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:50.092339 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:50.093295 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:50.093787 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:50.093970 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:50.094862 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:50.094867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:50.094880 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:50:50.095271 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:50.095325 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:50:50.095345 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:50.095457 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:50.095585 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 04:50:50.095829 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":148,"threads_started":1}
I20260808 04:50:50.097103 24438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:50.097908 24438 log.cc:824] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:50.099467 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:50:50.100160 24438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658: No bootstrap required, opened a new log
I20260808 04:50:50.103642 24438 raft_consensus.cc:348] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a768544b405a44c5a7c4b9b6fb07b658" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35367 } }
I20260808 04:50:50.103845 24438 raft_consensus.cc:374] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:50.103880 24438 raft_consensus.cc:729] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a768544b405a44c5a7c4b9b6fb07b658, State: Initialized, Role: FOLLOWER
I20260808 04:50:50.104213 24438 consensus_queue.cc:260] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a768544b405a44c5a7c4b9b6fb07b658" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35367 } }
I20260808 04:50:50.104319 24438 raft_consensus.cc:388] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:50.104362 24438 raft_consensus.cc:482] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:50.104506 24438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:50.105300 24438 raft_consensus.cc:504] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a768544b405a44c5a7c4b9b6fb07b658" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35367 } }
I20260808 04:50:50.105648 24438 leader_election.cc:302] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a768544b405a44c5a7c4b9b6fb07b658; no voters:
I20260808 04:50:50.106024 24438 leader_election.cc:288] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:50.106127 24444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:50.106428 24444 raft_consensus.cc:686] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [term 1 LEADER]: Becoming Leader. State: Replica: a768544b405a44c5a7c4b9b6fb07b658, State: Running, Role: LEADER
I20260808 04:50:50.106760 24444 consensus_queue.cc:237] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a768544b405a44c5a7c4b9b6fb07b658" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35367 } }
I20260808 04:50:50.107234 24438 sys_catalog.cc:564] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:50.108410 24445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a768544b405a44c5a7c4b9b6fb07b658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a768544b405a44c5a7c4b9b6fb07b658" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35367 } } }
I20260808 04:50:50.108564 24445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:50.108969 24461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:50.108911 24449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a768544b405a44c5a7c4b9b6fb07b658. Latest consensus state: current_term: 1 leader_uuid: "a768544b405a44c5a7c4b9b6fb07b658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a768544b405a44c5a7c4b9b6fb07b658" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35367 } } }
I20260808 04:50:50.109577 24449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:50.112317 24461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:50.117655 24461 catalog_manager.cc:1347] Generated new cluster ID: dcc135137f964801a378d279f8103144
I20260808 04:50:50.117756 24461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:50.138510 24461 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:50.139384 24461 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:50.152240 24461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a768544b405a44c5a7c4b9b6fb07b658: Generated new TSK 0
I20260808 04:50:50.153101 24461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:50:50.709678 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:50.169151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.15.30.126:35367)
0808 04:50:50.169293 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:50.169298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:50.169482 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:50:50.169685 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:50.169688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:50.169735 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:50.169844 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:50:50.170096 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:50.170602 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:50.171505 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:50.171730 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:50.171843 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:50.172158 (+ 315us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:50:50.709113 (+536955us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:50:50.709472 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43944: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 04:50:50.709784 24486 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:50.169079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35367 (local address 127.0.0.1:43944)
0808 04:50:50.169393 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:50.169405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:50.169534 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:50.169856 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:50.169859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:50.169867 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:50:50.169981 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:50.169992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:50.170737 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:50.170741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:50.171373 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:50.171381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:50.171856 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:50.171860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:50.171950 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:50.171982 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:50:50.709365 (+537383us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:50:50.709581 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.30.126:35367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
W20260808 04:50:50.709784 24441 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43944: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260808 04:50:50.710476 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 24321
2026-08-08T04:50:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (844 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (140 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:51 dist-test-slave-zpvx krb5kdc[24507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:51 dist-test-slave-zpvx krb5kdc[24507](info): set up 2 sockets
Aug 08 04:50:51 dist-test-slave-zpvx krb5kdc[24507](info): commencing operation
krb5kdc: starting...
W20260808 04:50:52.234308 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:52 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:50:52.358801 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:36325
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:52.470322 24526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:52.470588 24526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:52.470665 24526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:52.474618 24526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:52.474696 24526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:52.474733 24526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:52.474785 24526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:52.474828 24526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:52.479799 24526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:52.480791 24526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:52.481738 24526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:52.489981 24537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:52.490044 24526 server_base.cc:1034] running on GCE node
W20260808 04:50:52.490137 24542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:52.490028 24536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:52.490746 24526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:52.491715 24526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:52.492868 24526 hybrid_clock.cc:648] HybridClock initialized: now 1786164652492842 us; error 41 us; skew 500 ppm
Aug 08 04:50:52 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164652, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:52.495548 24526 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:50:52.496430 24526 webserver.cc:460] Webserver started at http://127.15.30.126:37687/ using document root <none> and password file <none>
I20260808 04:50:52.496968 24526 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:52.497033 24526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:52.497241 24526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:52.499213 24526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "089059a595d4496fba91ce063434a2f1"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "50a1f8ad7354eae993087b9efd307b47"
server_key_iv: "3e0808a08c4d563286d10f575976ac25"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.499764 24526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "089059a595d4496fba91ce063434a2f1"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "50a1f8ad7354eae993087b9efd307b47"
server_key_iv: "3e0808a08c4d563286d10f575976ac25"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.503460 24526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:52.505856 24554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:52.507228 24526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:52.507369 24526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "089059a595d4496fba91ce063434a2f1"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "50a1f8ad7354eae993087b9efd307b47"
server_key_iv: "3e0808a08c4d563286d10f575976ac25"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.507472 24526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:52.518517 24526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:52.521693 24526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:52.521905 24526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:52.529134 24526 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:36325
I20260808 04:50:52.529156 24639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:36325 every 8 connection(s)
I20260808 04:50:52.530054 24526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:52.532918 24640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:52.534833 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 24526
I20260808 04:50:52.534971 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:52.535322 15481 external_mini_cluster.cc:1428] Setting key 7a8bd287597ec0c3b92251b4d71a516d
I20260808 04:50:52.539710 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1: Bootstrap starting.
I20260808 04:50:52.542074 24640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:52.542829 24640 log.cc:824] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:50:52 dist-test-slave-zpvx krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:52.544950 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1: No bootstrap required, opened a new log
I20260808 04:50:52.548005 24640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089059a595d4496fba91ce063434a2f1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 36325 } }
I20260808 04:50:52.548275 24640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:52.548360 24640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089059a595d4496fba91ce063434a2f1, State: Initialized, Role: FOLLOWER
I20260808 04:50:52.548877 24640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089059a595d4496fba91ce063434a2f1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 36325 } }
I20260808 04:50:52.549055 24640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:52.549147 24640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:52.549254 24640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:52.550264 24640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089059a595d4496fba91ce063434a2f1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 36325 } }
I20260808 04:50:52.550617 24640 leader_election.cc:302] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089059a595d4496fba91ce063434a2f1; no voters:
I20260808 04:50:52.550958 24640 leader_election.cc:288] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:52.551072 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:52.536837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.15.30.126:36325)
0808 04:50:52.537317 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:52.537341 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:52.537381 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:50:52.538106 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:52.538125 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:52.538459 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:52.538832 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:52.539747 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:52.540934 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:52.541881 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:52.542456 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:52.542646 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:52.545394 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:52.545400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:52.545412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:52.545441 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:52.547745 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:52.548234 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:52.548239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:52.548243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:52.548483 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:52.548783 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:52.548787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:52.548789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:52.549317 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:52.549549 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:52.549873 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 04:50:52.550173 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":158,"threads_started":1}
I20260808 04:50:52.551072 24647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:52.551322 24647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [term 1 LEADER]: Becoming Leader. State: Replica: 089059a595d4496fba91ce063434a2f1, State: Running, Role: LEADER
I20260808 04:50:52.551640 24647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089059a595d4496fba91ce063434a2f1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 36325 } }
I20260808 04:50:52.552242 24640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:52.552865 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089059a595d4496fba91ce063434a2f1. Latest consensus state: current_term: 1 leader_uuid: "089059a595d4496fba91ce063434a2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089059a595d4496fba91ce063434a2f1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 36325 } } }
I20260808 04:50:52.553036 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:52.553498 24652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:52.552865 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "089059a595d4496fba91ce063434a2f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089059a595d4496fba91ce063434a2f1" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 36325 } } }
I20260808 04:50:52.554649 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:52.556833 24652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:52.562989 24652 catalog_manager.cc:1347] Generated new cluster ID: 5fc9ed81c7f34f668b53022317ec7938
I20260808 04:50:52.563052 24652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:52.581087 24652 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:52.581969 24652 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:52.590268 24652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089059a595d4496fba91ce063434a2f1: Generated new TSK 0
I20260808 04:50:52.590893 24652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:50:52.625648 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:36325
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:52.746405 24676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:52.746635 24676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:52.746676 24676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:52.750679 24676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:52.750733 24676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:52.750815 24676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:52.756228 24676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:52.757256 24676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:52.758266 24676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:50:52.768025 24676 server_base.cc:1034] running on GCE node
W20260808 04:50:52.767961 24683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:52.767961 24684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:52.767966 24687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:52.768642 24676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:52.769284 24676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:52.770473 24676 hybrid_clock.cc:648] HybridClock initialized: now 1786164652770460 us; error 27 us; skew 500 ppm
Aug 08 04:50:52 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164652, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:52.773380 24676 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:50:52.774214 24676 webserver.cc:460] Webserver started at http://127.15.30.65:39185/ using document root <none> and password file <none>
I20260808 04:50:52.774744 24676 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:52.774806 24676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:52.775038 24676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:52.777521 24676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "ef170011ecd04c6595613d1f736d4103"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "05a23d5ceb56e9de4a1d2bbe0af95cc2"
server_key_iv: "b47e58a0b762c2d7809ab64956b44e3f"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.778091 24676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "ef170011ecd04c6595613d1f736d4103"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "05a23d5ceb56e9de4a1d2bbe0af95cc2"
server_key_iv: "b47e58a0b762c2d7809ab64956b44e3f"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.781737 24676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:50:52.784031 24712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:52.785255 24676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:52.785374 24676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "ef170011ecd04c6595613d1f736d4103"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "05a23d5ceb56e9de4a1d2bbe0af95cc2"
server_key_iv: "b47e58a0b762c2d7809ab64956b44e3f"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.785475 24676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:52.797597 24676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:52.800176 24676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:52.800354 24676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:52.800994 24676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:52.801826 24676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:52.801874 24676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:52.801913 24676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:52.801954 24676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:52.810395 24676 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:35813
I20260808 04:50:52.810432 24910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:35813 every 8 connection(s)
I20260808 04:50:52.811318 24676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:52.811940 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 24676
I20260808 04:50:52.812072 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:52.812417 15481 external_mini_cluster.cc:1428] Setting key 2f881776c17cc3f46037019420d376e8
Aug 08 04:50:52 dist-test-slave-zpvx krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164652, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:50:52.825816 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:52.813327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:39535 (local address 127.15.30.126:36325)
0808 04:50:52.813457 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:52.813461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:52.814452 (+ 991us) server_negotiation.cc:407] Connection header received
0808 04:50:52.816476 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:52.816480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:52.816534 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:52.816634 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:52.817732 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:52.818259 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:52.819468 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:52.819700 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:52.819801 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:52.822158 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:52.822162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:52.822165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:52.822193 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:52.823612 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:52.824153 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:52.824156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:52.824157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:52.824234 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:52.824663 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:52.824670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:52.824675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:52.824928 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:52.825020 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:52.825558 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 04:50:52.825677 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:50:52.826635 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:52.813721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:36325 (local address 127.15.30.65:39535)
0808 04:50:52.814278 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:52.814334 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:52.816217 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:52.816808 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:52.816815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:52.817141 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:52.817586 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:52.817600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:52.818380 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:52.818384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:52.819341 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:52.819351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:52.819851 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:52.819855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:52.820019 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:52.820635 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:52.820650 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:52.822003 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:52.823769 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:52.823773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:52.823785 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:52.824033 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:52.824312 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:52.824314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:52.824316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:52.824542 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:52.825028 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:52.825032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:52.825273 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 04:50:52.825467 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 04:50:52.825833 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":150,"threads_started":1}
I20260808 04:50:52.827883 24913 heartbeater.cc:347] Connected to a master server at 127.15.30.126:36325
I20260808 04:50:52.828152 24913 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:52.828683 24913 heartbeater.cc:511] Master 127.15.30.126:36325 requested a full tablet report, sending...
I20260808 04:50:52.830207 24578 ts_manager.cc:194] Registered new tserver with Master: ef170011ecd04c6595613d1f736d4103 (127.15.30.65:35813)
I20260808 04:50:52.831331 24578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:39535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:50:52.842724 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:36325
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:52.951432 24925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:52.951709 24925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:52.951779 24925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:52.955960 24925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:52.956038 24925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:52.956127 24925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:52.961146 24925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:52.962103 24925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:52.963101 24925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:52.972313 24938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:52.972321 24937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:52.972340 24940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:52.972615 24925 server_base.cc:1034] running on GCE node
I20260808 04:50:52.972999 24925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:52.973631 24925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:52.974820 24925 hybrid_clock.cc:648] HybridClock initialized: now 1786164652974810 us; error 30 us; skew 500 ppm
Aug 08 04:50:52 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164652, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:52.977490 24925 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:50:52.978354 24925 webserver.cc:460] Webserver started at http://127.15.30.66:34967/ using document root <none> and password file <none>
I20260808 04:50:52.978883 24925 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:52.978971 24925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:52.979225 24925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:52.981202 24925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "b7d6a0b6837b48329ae7e187e5a6ed2d"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "3690958b816a3ab455a05ec70650061b"
server_key_iv: "a31dfd4ccbe4cfede344466cc5cb8064"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.981758 24925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "b7d6a0b6837b48329ae7e187e5a6ed2d"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "3690958b816a3ab455a05ec70650061b"
server_key_iv: "a31dfd4ccbe4cfede344466cc5cb8064"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.985325 24925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:50:52.987743 24955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:52.989192 24925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:50:52.989336 24925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "b7d6a0b6837b48329ae7e187e5a6ed2d"
format_stamp: "Formatted at 2026-08-08 04:50:52 on dist-test-slave-zpvx"
server_key: "3690958b816a3ab455a05ec70650061b"
server_key_iv: "a31dfd4ccbe4cfede344466cc5cb8064"
server_key_version: "encryptionkey@0"
I20260808 04:50:52.989440 24925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:53.011605 24925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:53.014810 24925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:53.014983 24925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:53.015575 24925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:53.016647 24925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:53.016705 24925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:53.016748 24925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:53.016770 24925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:53.025820 24925 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:38487
I20260808 04:50:53.025840 25131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:38487 every 8 connection(s)
I20260808 04:50:53.026688 24925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:53.028616 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 24925
I20260808 04:50:53.028775 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:53.029098 15481 external_mini_cluster.cc:1428] Setting key 1cbabfa1ab40109e7f8a74ed2c7a2c31
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164652, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:50:53.041389 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:53.028516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:39345 (local address 127.15.30.126:36325)
0808 04:50:53.028683 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.028689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.029630 (+ 941us) server_negotiation.cc:407] Connection header received
0808 04:50:53.031211 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.031213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.031255 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.031315 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.032985 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.033509 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.034654 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.034890 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.034999 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:53.037586 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:53.037591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:53.037595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:53.037622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:53.039412 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.039972 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.039975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.039976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.040052 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.040357 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.040359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.040360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.040626 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:53.040705 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:53.041163 (+ 458us) server_negotiation.cc:299] Negotiation successful
0808 04:50:53.041266 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:50:53.042250 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:53.028986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:36325 (local address 127.15.30.66:39345)
0808 04:50:53.029460 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:53.029512 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:53.030952 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:53.031505 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:53.031517 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:53.032072 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:53.032815 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:53.032841 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.033646 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.033650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:53.034529 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:53.034538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.035050 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.035055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:53.035276 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:53.035880 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:53.035895 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:53.037409 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:53.039563 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:53.039569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:53.039586 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:53.039862 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:53.040138 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:53.040141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:53.040142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:53.040270 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:53.040732 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:53.040737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:53.040954 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 04:50:53.041101 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 04:50:53.041349 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
I20260808 04:50:53.043392 25136 heartbeater.cc:347] Connected to a master server at 127.15.30.126:36325
I20260808 04:50:53.043672 25136 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:53.044323 25136 heartbeater.cc:511] Master 127.15.30.126:36325 requested a full tablet report, sending...
I20260808 04:50:53.045527 24578 ts_manager.cc:194] Registered new tserver with Master: b7d6a0b6837b48329ae7e187e5a6ed2d (127.15.30.66:38487)
I20260808 04:50:53.046128 24578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:39345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:50:53.058939 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:36325
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:53.169334 25143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:53.169552 25143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:53.169592 25143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:53.173480 25143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:53.173527 25143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:53.173588 25143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:53.178568 25143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:53.179464 25143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:53.180444 25143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:53.190079 25165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:53.190222 25143 server_base.cc:1034] running on GCE node
W20260808 04:50:53.190172 25155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:53.190088 25153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:53.190781 25143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:53.191356 25143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:53.192503 25143 hybrid_clock.cc:648] HybridClock initialized: now 1786164653192488 us; error 31 us; skew 500 ppm
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.195415 25143 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:50:53.196307 25143 webserver.cc:460] Webserver started at http://127.15.30.67:39391/ using document root <none> and password file <none>
I20260808 04:50:53.196911 25143 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:53.196977 25143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:53.197198 25143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:53.199344 25143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "175c1e36041e4327973310f4cc7a4971"
format_stamp: "Formatted at 2026-08-08 04:50:53 on dist-test-slave-zpvx"
server_key: "e722e7f45c5f0a5d3ae01d7753c70391"
server_key_iv: "aa0bb2f719dee84962a78d2f9029beab"
server_key_version: "encryptionkey@0"
I20260808 04:50:53.199916 25143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "175c1e36041e4327973310f4cc7a4971"
format_stamp: "Formatted at 2026-08-08 04:50:53 on dist-test-slave-zpvx"
server_key: "e722e7f45c5f0a5d3ae01d7753c70391"
server_key_iv: "aa0bb2f719dee84962a78d2f9029beab"
server_key_version: "encryptionkey@0"
I20260808 04:50:53.203723 25143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:50:53.206243 25178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:53.207458 25143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:53.207598 25143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "175c1e36041e4327973310f4cc7a4971"
format_stamp: "Formatted at 2026-08-08 04:50:53 on dist-test-slave-zpvx"
server_key: "e722e7f45c5f0a5d3ae01d7753c70391"
server_key_iv: "aa0bb2f719dee84962a78d2f9029beab"
server_key_version: "encryptionkey@0"
I20260808 04:50:53.207710 25143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:53.228327 25143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:53.231346 25143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:53.231539 25143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:53.232110 25143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:53.233083 25143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:53.233155 25143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:53.233270 25143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:53.233309 25143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:53.241997 25143 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:34631
I20260808 04:50:53.242026 25357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:34631 every 8 connection(s)
I20260808 04:50:53.242854 25143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:53.245049 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 25143
I20260808 04:50:53.245194 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:53.245565 15481 external_mini_cluster.cc:1428] Setting key cd08cdde7675207710ca375d79ed29bb
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:50:53.256910 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:53.244738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:49725 (local address 127.15.30.126:36325)
0808 04:50:53.244926 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.244933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.245807 (+ 874us) server_negotiation.cc:407] Connection header received
0808 04:50:53.247472 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.247474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.247517 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.247595 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.248865 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.249392 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.250535 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.250770 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.250847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:53.253434 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:53.253437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:53.253445 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:53.253467 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:53.254835 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.255367 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.255378 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.255379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.255456 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.255788 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.255793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.255795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.255966 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:53.256072 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:53.256608 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 04:50:53.256739 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:50:53.257774 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:53.245064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:36325 (local address 127.15.30.67:49725)
0808 04:50:53.245633 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:53.245697 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:53.247245 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:53.247792 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:53.247803 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:53.248184 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:53.248728 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:53.248743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.249530 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.249533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:53.250422 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:53.250433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.250903 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.250906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:53.251047 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:53.251656 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:53.251678 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:53.253279 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:53.254973 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:53.254979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:53.254992 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:53.255254 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:53.255546 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:53.255550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:53.255552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:53.255693 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:53.256071 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:53.256078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:53.256314 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 04:50:53.256565 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 04:50:53.256873 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":150,"threads_started":1}
I20260808 04:50:53.259117 25358 heartbeater.cc:347] Connected to a master server at 127.15.30.126:36325
I20260808 04:50:53.259382 25358 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:53.259876 25358 heartbeater.cc:511] Master 127.15.30.126:36325 requested a full tablet report, sending...
I20260808 04:50:53.260839 24577 ts_manager.cc:194] Registered new tserver with Master: 175c1e36041e4327973310f4cc7a4971 (127.15.30.67:34631)
I20260808 04:50:53.261404 24577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:49725
I20260808 04:50:53.271286 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/test-admin.keytab.
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.289495 15481 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 04:50:53.292555 25372 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:53.290947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.15.30.65:35813)
0808 04:50:53.291323 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.291338 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.291361 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:50:53.291407 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.291412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.291587 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.291735 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.292319 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59208: BlockingRecv error: recv got EOF from 127.0.0.1:59208 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
W20260808 04:50:53.292656 15481 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.293807 15481 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.294811 15481 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:53.304867 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:53.295592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.15.30.65:35813)
0808 04:50:53.295745 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.295748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.295763 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:53.295868 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.295870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.295911 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.295984 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.296491 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.297332 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.298224 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.298609 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.298709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:53.301116 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:53.301126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:53.301138 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:53.301162 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:53.302784 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.303373 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.303379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.303380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.303464 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.303783 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.303788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.303790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.304044 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:53.304290 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:53.304540 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 04:50:53.304724 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 04:50:53.308707 25372 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:53.307663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.15.30.65:35813)
0808 04:50:53.307787 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.307791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.307878 (+ 87us) server_negotiation.cc:407] Connection header received
0808 04:50:53.308019 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.308022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.308065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.308142 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.308588 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59234: BlockingRecv error: recv got EOF from 127.0.0.1:59234 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 04:50:53.308856 15481 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.309938 15481 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.310885 15481 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:53.319577 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:53.311758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.15.30.65:35813)
0808 04:50:53.311868 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.311872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.312071 (+ 199us) server_negotiation.cc:407] Connection header received
0808 04:50:53.312193 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.312195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.312228 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.312291 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.312891 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.313415 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.314209 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.314422 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.314499 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:53.316756 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:53.316758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:53.316760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:53.316780 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:53.318173 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.318555 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.318557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.318559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.318628 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.318862 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.318864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.318865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.319003 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:53.319092 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:53.319302 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 04:50:53.319417 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 04:50:53.322176 25372 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:50:53.321231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.15.30.65:35813)
0808 04:50:53.321357 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.321361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.321452 (+ 91us) server_negotiation.cc:407] Connection header received
0808 04:50:53.321539 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.321541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.321572 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.321639 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.322075 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59252: BlockingRecv error: recv got EOF from 127.0.0.1:59252 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:50:53.322204 15481 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786164620544947-15481-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.323235 15481 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:53.324147 15481 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:50:53.333071 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:53.325016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.15.30.65:35813)
0808 04:50:53.325159 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:53.325162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:53.325193 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:50:53.325316 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:53.325318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:53.325372 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:53.325466 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:53.325941 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.326453 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.327228 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:53.327559 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:53.327655 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:53.329837 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:53.329840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:53.329842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:53.329864 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:53.331361 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.331742 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.331745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.331746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.331821 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:53.332104 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:53.332106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:53.332108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:53.332269 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:53.332426 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:53.332773 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 04:50:53.332902 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:50:53.334095 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 24676
I20260808 04:50:53.348701 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 24925
I20260808 04:50:53.372076 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 25143
I20260808 04:50:53.385674 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 24526
2026-08-08T04:50:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2412 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[25395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[25395](info): set up 2 sockets
Aug 08 04:50:53 dist-test-slave-zpvx krb5kdc[25395](info): commencing operation
krb5kdc: starting...
W20260808 04:50:54.639056 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:50:54 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:50:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab.
Entry for principal oryx/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab.
I20260808 04:50:54.768285 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34745
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab
--principal=oryx/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:34745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:54.878360 25419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:54.878595 25419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:54.878635 25419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:54.882454 25419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:54.882500 25419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:54.882519 25419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:54.882535 25419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:54.882550 25419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:54.887499 25419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:34745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab
--principal=oryx/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:54.888427 25419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:54.889369 25419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:54.897688 25430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:54.897738 25427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:54.897799 25428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:54.898087 25419 server_base.cc:1034] running on GCE node
I20260808 04:50:54.898597 25419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:54.899607 25419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:54.900764 25419 hybrid_clock.cc:648] HybridClock initialized: now 1786164654900745 us; error 31 us; skew 500 ppm
Aug 08 04:50:54 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164654, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:54.907114 25419 init.cc:377] Logged in from keytab as oryx/127.15.30.126@KRBTEST.COM (short username oryx)
I20260808 04:50:54.907994 25419 webserver.cc:460] Webserver started at http://127.15.30.126:42385/ using document root <none> and password file <none>
I20260808 04:50:54.908565 25419 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:54.908628 25419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:54.908795 25419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:54.910738 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "f5a1c13a8d4244699977da3aceb3a1da"
format_stamp: "Formatted at 2026-08-08 04:50:54 on dist-test-slave-zpvx"
server_key: "aeca511edd9fcc2c0c4e7d06e9645062"
server_key_iv: "3026d036f63352a6dab22b0255c06287"
server_key_version: "encryptionkey@0"
I20260808 04:50:54.911259 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "f5a1c13a8d4244699977da3aceb3a1da"
format_stamp: "Formatted at 2026-08-08 04:50:54 on dist-test-slave-zpvx"
server_key: "aeca511edd9fcc2c0c4e7d06e9645062"
server_key_iv: "3026d036f63352a6dab22b0255c06287"
server_key_version: "encryptionkey@0"
I20260808 04:50:54.914937 25419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260808 04:50:54.917328 25438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:54.918532 25419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:50:54.918668 25419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "f5a1c13a8d4244699977da3aceb3a1da"
format_stamp: "Formatted at 2026-08-08 04:50:54 on dist-test-slave-zpvx"
server_key: "aeca511edd9fcc2c0c4e7d06e9645062"
server_key_iv: "3026d036f63352a6dab22b0255c06287"
server_key_version: "encryptionkey@0"
I20260808 04:50:54.918773 25419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:54.933398 25419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:54.936439 25419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:54.936625 25419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:54.943415 25419 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:34745
I20260808 04:50:54.943698 25522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:34745 every 8 connection(s)
I20260808 04:50:54.944350 25419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:50:54.947242 25524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:54.953189 25524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: Bootstrap starting.
I20260808 04:50:54.954398 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 25419
I20260808 04:50:54.954552 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:50:54.954838 15481 external_mini_cluster.cc:1428] Setting key 84e07b34f7b5e6062664572cc34e7a48
I20260808 04:50:54.956131 25524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:54.956995 25524 log.cc:824] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:54.958922 25524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: No bootstrap required, opened a new log
I20260808 04:50:54.962741 25524 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } }
I20260808 04:50:54.962996 25524 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:54.963084 25524 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a1c13a8d4244699977da3aceb3a1da, State: Initialized, Role: FOLLOWER
Aug 08 04:50:54 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:50:54.963539 25524 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } }
I20260808 04:50:54.963708 25524 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:50:54.963785 25524 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:50:54.963918 25524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:54.965097 25524 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } }
I20260808 04:50:54.965451 25524 leader_election.cc:302] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a1c13a8d4244699977da3aceb3a1da; no voters:
I20260808 04:50:54.965787 25524 leader_election.cc:288] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:50:54.965858 25535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:54.966101 25535 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 1 LEADER]: Becoming Leader. State: Replica: f5a1c13a8d4244699977da3aceb3a1da, State: Running, Role: LEADER
I20260808 04:50:54.966465 25535 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } }
I20260808 04:50:54.967104 25524 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:54.968466 25536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } }
I20260808 04:50:54.968604 25536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:54.969475 25537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a1c13a8d4244699977da3aceb3a1da. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } }
I20260808 04:50:54.969609 25537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:54.969615 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:54.955906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.15.30.126:34745)
0808 04:50:54.956299 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:54.956315 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:54.956349 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:50:54.957003 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:54.957019 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:54.957385 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:54.957707 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:54.958508 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:54.959520 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:54.960346 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:54.960859 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:54.961031 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:54.963781 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:54.963790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:54.963811 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:54.963841 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:54.965796 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:54.966364 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:54.966370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:54.966375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:54.966509 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:54.966822 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:54.966826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:54.966828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:54.967310 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:54.967482 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:54.968339 (+ 857us) server_negotiation.cc:299] Negotiation successful
0808 04:50:54.968648 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260808 04:50:54.970118 25546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:54.972944 25546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:54.978482 25546 catalog_manager.cc:1347] Generated new cluster ID: eb89cc6dc32e401bbf5d3acafbe85a83
I20260808 04:50:54.978547 25546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:54.988731 25546 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:50:54.989671 25546 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:54.998097 25546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: Generated new TSK 0
I20260808 04:50:54.998723 25546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab.
Entry for principal oryx/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab.
I20260808 04:50:55.031057 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34745
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab
--principal=oryx/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:55.141788 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:55.142055 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:55.142124 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:55.146107 25568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:55.146183 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:55.146274 25568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:55.151298 25568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab
--principal=oryx/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:55.152313 25568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:55.153339 25568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:55.163017 25581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:55.163072 25579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:55.163017 25578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:55.163290 25568 server_base.cc:1034] running on GCE node
I20260808 04:50:55.163749 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:55.164364 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:55.165606 25568 hybrid_clock.cc:648] HybridClock initialized: now 1786164655165576 us; error 48 us; skew 500 ppm
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:55.168478 25568 init.cc:377] Logged in from keytab as oryx/127.15.30.65@KRBTEST.COM (short username oryx)
I20260808 04:50:55.169329 25568 webserver.cc:460] Webserver started at http://127.15.30.65:46455/ using document root <none> and password file <none>
I20260808 04:50:55.169878 25568 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:55.169967 25568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:55.170192 25568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:55.172138 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "2eaf5c3842aa4ad28cfae11225fd345c"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "5c120f0508e2d074344842bf5d981cfe"
server_key_iv: "ed2069aa9765e6277be0b19f1fca769d"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.172730 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "2eaf5c3842aa4ad28cfae11225fd345c"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "5c120f0508e2d074344842bf5d981cfe"
server_key_iv: "ed2069aa9765e6277be0b19f1fca769d"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.176232 25568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:50:55.178557 25594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.179754 25568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:55.179879 25568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "2eaf5c3842aa4ad28cfae11225fd345c"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "5c120f0508e2d074344842bf5d981cfe"
server_key_iv: "ed2069aa9765e6277be0b19f1fca769d"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.179980 25568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:55.198159 25568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:55.200968 25568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:55.201156 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:55.201715 25568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:55.202495 25568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:55.202562 25568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.202634 25568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:55.202682 25568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.211582 25568 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:40717
I20260808 04:50:55.211611 25784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:40717 every 8 connection(s)
I20260808 04:50:55.212450 25568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:55.217536 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 25568
I20260808 04:50:55.217639 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:50:55.217928 15481 external_mini_cluster.cc:1428] Setting key 7638252f22c8fa5e1e62689577b236d4
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:50:55.227308 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.214300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:44747 (local address 127.15.30.126:34745)
0808 04:50:55.214453 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.214458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.215326 (+ 868us) server_negotiation.cc:407] Connection header received
0808 04:50:55.216719 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.216722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.216769 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.216845 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.218072 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.218792 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.220467 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.220702 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.220783 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.223557 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.223561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.223564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.223597 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.225181 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.225736 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.225740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.225742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.225829 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.226206 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.226209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.226211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.226401 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.226509 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.227001 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.227133 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:50:55.228104 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.214715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.65:44747)
0808 04:50:55.215168 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.215217 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.216507 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.217004 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.217011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.217359 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.217904 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.217917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.218920 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.218928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.220256 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.220271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.220858 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.220865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.221101 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.221865 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.221881 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.223384 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.225310 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.225316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.225331 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.225618 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.225925 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.225928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.225930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.226086 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.226509 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.226517 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.226767 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.226958 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.227292 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for oryx/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:50:55.229480 25785 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34745
I20260808 04:50:55.229710 25785 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:55.230175 25785 heartbeater.cc:511] Master 127.15.30.126:34745 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.66@KRBTEST.COM" created.
I20260808 04:50:55.231807 25464 ts_manager.cc:194] Registered new tserver with Master: 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717)
I20260808 04:50:55.233110 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:44747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab.
Entry for principal oryx/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab.
I20260808 04:50:55.248353 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34745
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab
--principal=oryx/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:55.358103 25791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:55.358325 25791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:55.358364 25791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:55.362322 25791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:55.362373 25791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:55.362455 25791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:55.367549 25791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab
--principal=oryx/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:55.368672 25791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:55.369704 25791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:55.378858 25808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:55.378881 25802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:55.378863 25791 server_base.cc:1034] running on GCE node
W20260808 04:50:55.378862 25801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:55.379520 25791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:55.380075 25791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:55.381229 25791 hybrid_clock.cc:648] HybridClock initialized: now 1786164655381217 us; error 25 us; skew 500 ppm
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:55.383867 25791 init.cc:377] Logged in from keytab as oryx/127.15.30.66@KRBTEST.COM (short username oryx)
I20260808 04:50:55.384718 25791 webserver.cc:460] Webserver started at http://127.15.30.66:32787/ using document root <none> and password file <none>
I20260808 04:50:55.385198 25791 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:55.385253 25791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:55.385406 25791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:55.387270 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "88b1338766d540a584b8fd50b4e950b9"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "bd79ab533dc3e9592f98109aef1fbee8"
server_key_iv: "e6eff8657afb182790faeaaf6dda1366"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.387765 25791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "88b1338766d540a584b8fd50b4e950b9"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "bd79ab533dc3e9592f98109aef1fbee8"
server_key_iv: "e6eff8657afb182790faeaaf6dda1366"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.391273 25791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:55.393558 25821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.394774 25791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:55.394899 25791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "88b1338766d540a584b8fd50b4e950b9"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "bd79ab533dc3e9592f98109aef1fbee8"
server_key_iv: "e6eff8657afb182790faeaaf6dda1366"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.395000 25791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:55.408573 25791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:55.411188 25791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:55.411386 25791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:55.411892 25791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:55.412753 25791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:55.412818 25791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.412881 25791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:55.412927 25791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.421613 25791 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:40251
I20260808 04:50:55.421640 25989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:40251 every 8 connection(s)
I20260808 04:50:55.422437 25791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:55.424604 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 25791
I20260808 04:50:55.424721 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:50:55.425061 15481 external_mini_cluster.cc:1428] Setting key 9753817917e9c37305b23ab0c53594c2
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:50:55.438207 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.424240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:59063 (local address 127.15.30.126:34745)
0808 04:50:55.424411 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.424419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.425294 (+ 875us) server_negotiation.cc:407] Connection header received
0808 04:50:55.426949 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.426952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.426992 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.427051 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.428692 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.429571 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.431115 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.431326 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.431406 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.434503 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.434506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.434509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.434537 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.435946 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.436491 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.436494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.436495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.436625 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.437134 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.437136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.437137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.437303 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.437378 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.437972 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.438072 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:50:55.438997 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.424568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.66:59063)
0808 04:50:55.425120 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.425180 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.426718 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.427230 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.427241 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.427777 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.428505 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.428529 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.429714 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.429717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.430964 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.430980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.431481 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.431488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.431772 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.432536 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.432562 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.434339 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.436074 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.436080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.436093 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.436353 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.436797 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.436804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.436806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.437026 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.437420 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.437426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.437669 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.437877 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.438128 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":160,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.67@KRBTEST.COM" created.
I20260808 04:50:55.440225 25992 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34745
I20260808 04:50:55.440502 25992 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:55.441092 25992 heartbeater.cc:511] Master 127.15.30.126:34745 requested a full tablet report, sending...
I20260808 04:50:55.442282 25464 ts_manager.cc:194] Registered new tserver with Master: 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251)
I20260808 04:50:55.442835 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:59063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab.
Entry for principal oryx/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab.
I20260808 04:50:55.455788 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34745
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab
--principal=oryx/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:55.565397 26000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:55.565615 26000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:55.565676 26000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:55.569902 26000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:55.569952 26000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:55.570036 26000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:55.575021 26000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab
--principal=oryx/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:55.575954 26000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:55.576915 26000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:55.586381 26011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:55.586452 26009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:55.586381 26008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:55.586567 26000 server_base.cc:1034] running on GCE node
I20260808 04:50:55.586969 26000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:55.587558 26000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:55.588733 26000 hybrid_clock.cc:648] HybridClock initialized: now 1786164655588718 us; error 27 us; skew 500 ppm
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:55.591527 26000 init.cc:377] Logged in from keytab as oryx/127.15.30.67@KRBTEST.COM (short username oryx)
I20260808 04:50:55.592341 26000 webserver.cc:460] Webserver started at http://127.15.30.67:36067/ using document root <none> and password file <none>
I20260808 04:50:55.592906 26000 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:55.592969 26000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:55.593130 26000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:55.595052 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "cd1a534a60b44186a8a079e9d4c4c0cc"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "907459356c98fd4b58fe33c7dcc22c58"
server_key_iv: "4ff0f88e2a038fb11a7d6ddcfa71550f"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.595553 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "cd1a534a60b44186a8a079e9d4c4c0cc"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "907459356c98fd4b58fe33c7dcc22c58"
server_key_iv: "4ff0f88e2a038fb11a7d6ddcfa71550f"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.599248 26000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 04:50:55.601652 26025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.602779 26000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:55.602872 26000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "cd1a534a60b44186a8a079e9d4c4c0cc"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "907459356c98fd4b58fe33c7dcc22c58"
server_key_iv: "4ff0f88e2a038fb11a7d6ddcfa71550f"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.602950 26000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:55.620906 26000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:55.623924 26000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:55.624114 26000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:55.624835 26000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:55.625571 26000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:55.625618 26000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.625655 26000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:55.625684 26000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.634783 26000 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:46127
I20260808 04:50:55.634830 26191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:46127 every 8 connection(s)
I20260808 04:50:55.635599 26000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:55.641712 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 26000
I20260808 04:50:55.641835 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:50:55.642122 15481 external_mini_cluster.cc:1428] Setting key ba5e731f46b2d76172d419edf6e80672
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:50:55.649853 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.637401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:43313 (local address 127.15.30.126:34745)
0808 04:50:55.637548 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.637553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.638453 (+ 900us) server_negotiation.cc:407] Connection header received
0808 04:50:55.639936 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.639940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.640005 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.640134 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.641338 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.642238 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.643766 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.644002 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.644203 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.646553 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.646556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.646558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.646581 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.647808 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.648321 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.648325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.648327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.648458 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.648801 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.648805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.648807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.648981 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.649098 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.649568 (+ 470us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.649686 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:50:55.650581 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.637733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:43313)
0808 04:50:55.638198 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.638241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.639710 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.640312 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.640319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.640703 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.641182 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.641196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.642368 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.642371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.643616 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.643627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.644241 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.644246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.644472 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.644958 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.644974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.646387 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.647934 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.647940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.647953 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.648217 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.648565 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.648568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.648570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.648705 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.649107 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.649116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.649334 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.649523 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.649826 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":121,"threads_started":1}
I20260808 04:50:55.651672 26192 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34745
I20260808 04:50:55.651868 26192 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:55.652268 26192 heartbeater.cc:511] Master 127.15.30.126:34745 requested a full tablet report, sending...
I20260808 04:50:55.653275 25464 ts_manager.cc:194] Registered new tserver with Master: cd1a534a60b44186a8a079e9d4c4c0cc (127.15.30.67:46127)
I20260808 04:50:55.653818 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:43313
I20260808 04:50:55.655928 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.30.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.125.keytab.
Entry for principal oryx/127.15.30.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.125.keytab.
I20260808 04:50:55.685024 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:34027
--webserver_interface=127.15.30.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.125.keytab
--principal=oryx/127.15.30.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:34745,127.15.30.125:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:50:55.798197 26199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:55.798453 26199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:55.798524 26199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:55.802500 26199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:55.802578 26199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:55.802614 26199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:55.802652 26199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:55.802688 26199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:55.807766 26199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:34745,127.15.30.125:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.125.keytab
--principal=oryx/127.15.30.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.30.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
W20260808 04:50:55.807996 26199 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.30.126:34745,127.15.30.125:34027'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 04:50:55.808823 26199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:55.809801 26199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:55.818007 26207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:55.818013 26206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:55.818166 26199 server_base.cc:1034] running on GCE node
W20260808 04:50:55.818322 26210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:55.818941 26199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:55.819924 26199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:55.821080 26199 hybrid_clock.cc:648] HybridClock initialized: now 1786164655821061 us; error 32 us; skew 500 ppm
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:55.823850 26199 init.cc:377] Logged in from keytab as oryx/127.15.30.125@KRBTEST.COM (short username oryx)
I20260808 04:50:55.824733 26199 webserver.cc:460] Webserver started at http://127.15.30.125:40433/ using document root <none> and password file <none>
I20260808 04:50:55.825266 26199 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:55.825353 26199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:55.825568 26199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:55.827499 26199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/instance:
uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "0b5102cb5428f44dc1616ec6ed5658a7"
server_key_iv: "e3680d054d7a280c1fa2753908bfe862"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.828053 26199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal/instance:
uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "0b5102cb5428f44dc1616ec6ed5658a7"
server_key_iv: "e3680d054d7a280c1fa2753908bfe862"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.831660 26199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:50:55.834002 26223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.835137 26199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:55.835281 26199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "0b5102cb5428f44dc1616ec6ed5658a7"
server_key_iv: "e3680d054d7a280c1fa2753908bfe862"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.835386 26199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:55.850839 26199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:55.853497 26199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:55.853681 26199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:55.860210 26199 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.125:34027
I20260808 04:50:55.860530 26304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.125:34027 every 8 connection(s)
I20260808 04:50:55.861172 26199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
I20260808 04:50:55.863773 26305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:55.867424 26305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:50:55.871030 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 26199
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.125@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.30.125@KRBTEST.COM
I20260808 04:50:55.882078 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.869397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.15.30.126:34745)
0808 04:50:55.869526 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.869530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.870304 (+ 774us) server_negotiation.cc:407] Connection header received
0808 04:50:55.871303 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.871306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.871359 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.871433 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.873072 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.873614 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.874743 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.874973 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.875053 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.877992 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.877999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.878006 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.878040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.879518 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.880280 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.880285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.880287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.880443 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.880828 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.880830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.880831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.881043 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.881122 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.881742 (+ 620us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.881879 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:50:55.883000 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.869728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:36918)
0808 04:50:55.870149 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.870189 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.871067 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.871631 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.871638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.872048 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.872885 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.872901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.873766 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.873769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.874600 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.874610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.875125 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.875129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.875328 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.875942 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.875958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.877814 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.879639 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.879652 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.879662 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.880183 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.880536 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.880541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.880543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.880731 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.881155 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.881164 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.881464 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.881660 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.881961 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260808 04:50:55.883061 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.872020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.15.30.125:34027)
0808 04:50:55.872513 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.872522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.872543 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:50:55.872711 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.872716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.872857 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.873008 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.873745 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.874628 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.875719 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.876061 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.876177 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.879231 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.879237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.879247 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.879268 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.881228 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.881651 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.881657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.881658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.881746 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.882039 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.882042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.882044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.882306 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.882500 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.882751 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.882890 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":407,"thread_start_us":108,"threads_started":1}
I20260808 04:50:55.885079 26305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164655, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.125@KRBTEST.COM for oryx/127.15.30.125@KRBTEST.COM
I20260808 04:50:55.892619 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.885560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.0.0.1:55462)
0808 04:50:55.885669 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.885685 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.885759 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.885943 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.885945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.886076 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.886160 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.886165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.886906 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.886909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.887403 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.887411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.887859 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.887862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.887939 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.888307 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.888318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.889591 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.891215 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.891218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.891221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.891404 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.891671 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.891674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.891676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.891751 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.892090 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.892092 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.892182 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.892281 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.892449 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:50:55.892637 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.885705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.15.30.125:34027)
0808 04:50:55.885801 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.885804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.885814 (+ 10us) server_negotiation.cc:407] Connection header received
0808 04:50:55.885841 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.885843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.885874 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.885935 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.886250 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.886788 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.887515 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.887746 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.887824 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.889722 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.889726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.889728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.889748 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.891051 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.891500 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.891505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.891507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.891579 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.891841 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.891844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.891846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.892001 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.892100 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.892348 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.892507 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":36}
I20260808 04:50:55.895887 26305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Bootstrap starting.
I20260808 04:50:55.898285 26305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:55.899058 26305 log.cc:824] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:55.900954 26305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: No bootstrap required, opened a new log
I20260808 04:50:55.903810 26305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } }
I20260808 04:50:55.903992 26305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:55.904068 26305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ce5b0398a04b92b2311e6ab9f2aa5a, State: Initialized, Role: FOLLOWER
I20260808 04:50:55.904507 26305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } }
I20260808 04:50:55.905025 26313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } } }
I20260808 04:50:55.905161 26313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:50:55.905793 26305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:50:55.910673 26327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:50:55.910763 26327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:50:55.912889 26199 master_runner.cc:428] Detected that this master 03ce5b0398a04b92b2311e6ab9f2aa5a is joining an existing cluster
I20260808 04:50:55.912953 26199 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.30.125:34027
I20260808 04:50:55.921641 26313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:55.921790 26313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } }
I20260808 04:50:55.923403 26313 leader_election.cc:288] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745)
I20260808 04:50:55.923928 25476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a1c13a8d4244699977da3aceb3a1da" is_pre_election: true
I20260808 04:50:55.924337 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.915288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.15.30.125:34027)
0808 04:50:55.915414 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.915419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.915977 (+ 558us) server_negotiation.cc:407] Connection header received
0808 04:50:55.916066 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.916069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.916113 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.916150 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.916984 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.917756 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.918620 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.918904 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.919004 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.920351 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.920355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.920357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.920406 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.922295 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.922893 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.922897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.922901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.922991 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.923281 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.923284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.923286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.923553 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.923657 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.924026 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.924202 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:50:55.924896 26224 leader_election.cc:302] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 03ce5b0398a04b92b2311e6ab9f2aa5a; no voters: f5a1c13a8d4244699977da3aceb3a1da
I20260808 04:50:55.925194 26313 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:50:55.924955 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.915299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.0.0.1:55468)
0808 04:50:55.915837 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.915853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.915971 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.916279 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.916284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.916625 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.916859 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.916868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.917890 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.917893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.918493 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.918501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.919033 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.919039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.919139 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.919743 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.919758 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.920217 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.922454 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.922459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.922466 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.922731 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.923098 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.923100 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.923102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.923187 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.923723 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.923725 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.923849 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.924630 (+ 781us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.924779 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":22,"spinlock_wait_cycles":5120,"thread_start_us":97,"threads_started":1}
I20260808 04:50:55.925689 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.915416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.15.30.126:34745)
0808 04:50:55.915531 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.915535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.915941 (+ 406us) server_negotiation.cc:407] Connection header received
0808 04:50:55.916070 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.916074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.916124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.916223 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.917899 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.918700 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.920312 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.920629 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.920746 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.921948 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.921952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.921955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.921982 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.923685 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.924350 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.924355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.924357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.924471 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.924771 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.924775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.924776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.924966 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.925032 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.925436 (+ 404us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.925547 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 04:50:55.926103 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.915521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:36932)
0808 04:50:55.915849 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.915862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.915971 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.916247 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.916250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.916629 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.916859 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.916868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.918805 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.918808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.920183 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.920194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.920738 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.920742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.920872 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.921305 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.921319 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.921776 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.923918 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.923922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.923924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.924233 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.924587 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.924590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.924592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.924672 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.925104 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.925107 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.925213 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.925806 (+ 593us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.925937 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":46,"spinlock_wait_cycles":9600,"thread_start_us":119,"threads_started":1}
Aug 08 04:50:55 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164654, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.126@KRBTEST.COM for oryx/127.15.30.125@KRBTEST.COM
I20260808 04:50:55.939357 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.930110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.15.30.125:34027)
0808 04:50:55.930224 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.930228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.930527 (+ 299us) server_negotiation.cc:407] Connection header received
0808 04:50:55.930739 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.930742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.930791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.930862 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.931438 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.932143 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.933044 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.933378 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.933479 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.935554 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.935558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.935559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.935586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.937283 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.937907 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.937912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.937917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.938005 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.938363 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.938369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.938373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.938611 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.938715 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.939107 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.939215 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:50:55.939373 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.930066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.0.0.1:55478)
0808 04:50:55.930406 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.930451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.930595 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.930912 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.930917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.931238 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.931333 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.931338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.932266 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.932269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.932932 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.932942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.933480 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.933482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.933556 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.934042 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.934053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.935414 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.937420 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.937433 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.937442 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.937774 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.938114 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.938120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.938124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.938221 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.938779 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.938789 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.938900 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.939045 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.939220 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
I20260808 04:50:55.941062 25464 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.30.125:34027: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a"
member_type: NON_VOTER
last_known_addr {
host: "127.15.30.125"
port: 34027
}
attrs {
promote: true
}
}
dest_uuid: "f5a1c13a8d4244699977da3aceb3a1da"
cas_config_opid_index: -1
I20260808 04:50:55.941694 25464 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } }
I20260808 04:50:55.943565 26344 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 03ce5b0398a04b92b2311e6ab9f2aa5a (127.15.30.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } }
I20260808 04:50:55.943958 26266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:55.945761 26344 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.30.125:34027
I20260808 04:50:55.945891 26266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Refusing update from remote peer f5a1c13a8d4244699977da3aceb3a1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 04:50:55.946349 26344 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } }
I20260808 04:50:55.946420 26342 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:55.946548 26344 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.15.30.124@KRBTEST.COM; defaulting to no policy
I20260808 04:50:55.951666 26266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 03ce5b0398a04b92b2311e6ab9f2aa5a (127.15.30.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } }
I20260808 04:50:55.952121 26348 mvcc.cc:205] Tried to move back new op lower bound from 7316130426868363264 to 7316130426743554048. Current Snapshot: MvccSnapshot[applied={T|T < 7316130426782408704}]
I20260808 04:50:55.953858 26313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a1c13a8d4244699977da3aceb3a1da. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } }
I20260808 04:50:55.953963 26313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEARNER
I20260808 04:50:55.954052 26313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } }
I20260808 04:50:55.954156 26313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEARNER
I20260808 04:50:55.954233 26313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } }
I20260808 04:50:55.954294 26313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEARNER
I20260808 04:50:55.956256 26199 master.cc:537] Master@127.15.30.125:34027 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.124@KRBTEST.COM" created.
I20260808 04:50:55.962682 26199 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Raft consensus shutting down.
I20260808 04:50:55.963039 26199 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Raft consensus is shut down!
I20260808 04:50:55.963174 26199 tablet_replica.cc:330] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: stopping tablet replica
I20260808 04:50:55.968745 26199 master.cc:552] Master@127.15.30.125:34027 shutdown complete.
I20260808 04:50:55.968828 26199 master_runner.cc:305] Clearing existing system tablet
I20260808 04:50:55.969146 26199 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:55.969236 26199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:55.970808 26199 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:55.971982 26364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:55.972205 26199 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:55.972340 26199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:55.972414 26199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "0b5102cb5428f44dc1616ec6ed5658a7"
server_key_iv: "e3680d054d7a280c1fa2753908bfe862"
server_key_version: "encryptionkey@0"
I20260808 04:50:55.974027 26199 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:55.976610 26199 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:50:55.976713 26199 log.cc:1196] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 04:50:55.977033 26199 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Deleting consensus metadata
I20260808 04:50:55.977949 26199 master_runner.cc:315] Copying system tablet from 127.15.30.126:34745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.124.keytab.
Entry for principal oryx/127.15.30.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.124.keytab.
I20260808 04:50:55.979121 26199 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.30.126:34745
I20260808 04:50:55.979625 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:40143
--webserver_interface=127.15.30.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.124.keytab
--principal=oryx/127.15.30.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:50:55.988027 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.979758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.15.30.126:34745)
0808 04:50:55.979914 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:55.979921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:55.980361 (+ 440us) server_negotiation.cc:407] Connection header received
0808 04:50:55.980514 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:55.980517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:55.980570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:55.980657 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:55.981273 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.981822 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.982630 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.982851 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.982974 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.984284 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:55.984289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:55.984291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:55.984318 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:55.986081 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.986610 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.986614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.986616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.986703 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:55.987044 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:55.987047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:55.987049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:55.987304 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:55.987402 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:55.987750 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 04:50:55.987868 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:50:55.988157 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:55.979972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:36936)
0808 04:50:55.980248 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:55.980266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:55.980414 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:55.980694 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:55.980700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:55.980932 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:55.981102 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.981114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.981938 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.981940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.982503 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:55.982509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:55.982962 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:55.982966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:55.983078 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:55.983639 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:55.983659 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:55.984156 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:55.986203 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.986210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.986213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.986506 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.986806 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:55.986812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:55.986817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:55.986927 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:55.987412 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:55.987419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:55.987547 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 04:50:55.987674 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 04:50:55.987855 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260808 04:50:55.989096 25495 tablet_copy_service.cc:140] P f5a1c13a8d4244699977da3aceb3a1da: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 03ce5b0398a04b92b2311e6ab9f2aa5a ({username='oryx', principal='oryx/127.15.30.125@KRBTEST.COM'} at 127.0.0.1:36936)
I20260808 04:50:55.989248 25495 tablet_copy_service.cc:161] P f5a1c13a8d4244699977da3aceb3a1da: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 03ce5b0398a04b92b2311e6ab9f2aa5a at {username='oryx', principal='oryx/127.15.30.125@KRBTEST.COM'} at 127.0.0.1:36936: session id = 03ce5b0398a04b92b2311e6ab9f2aa5a-00000000000000000000000000000000
I20260808 04:50:55.991017 25495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:50:55.992784 26199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:50:55.994974 26199 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: tablet copy: Starting download of 0 data blocks...
I20260808 04:50:55.995144 26199 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: tablet copy: Starting download of 1 WAL segments...
I20260808 04:50:55.996865 26199 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:50:55.998728 25495 tablet_copy_service.cc:342] P f5a1c13a8d4244699977da3aceb3a1da: Request end of tablet copy session 03ce5b0398a04b92b2311e6ab9f2aa5a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.30.125@KRBTEST.COM'} at 127.0.0.1:36936
I20260808 04:50:55.998837 25495 tablet_copy_service.cc:433] P f5a1c13a8d4244699977da3aceb3a1da: ending tablet copy session 03ce5b0398a04b92b2311e6ab9f2aa5a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 03ce5b0398a04b92b2311e6ab9f2aa5a
I20260808 04:50:56.006132 26199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:56.007920 26375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.008067 26379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.008082 26376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.008344 26199 server_base.cc:1034] running on GCE node
I20260808 04:50:56.008661 26199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:56.008945 26199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:56.010097 26199 hybrid_clock.cc:648] HybridClock initialized: now 1786164656010075 us; error 39 us; skew 500 ppm
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:56.011739 26199 init.cc:377] Logged in from keytab as oryx/127.15.30.125@KRBTEST.COM (short username oryx)
I20260808 04:50:56.012190 26199 webserver.cc:460] Webserver started at http://127.15.30.125:33365/ using document root <none> and password file <none>
I20260808 04:50:56.012452 26199 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:56.012507 26199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:56.013922 26199 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:56.014921 26393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.015142 26199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:56.015209 26199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "0b5102cb5428f44dc1616ec6ed5658a7"
server_key_iv: "e3680d054d7a280c1fa2753908bfe862"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.015370 26199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:56.052902 26199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:56.055269 26199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:56.060410 26199 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.125:34027
I20260808 04:50:56.060527 26481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.125:34027 every 8 connection(s)
I20260808 04:50:56.061029 26199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
I20260808 04:50:56.062979 26483 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:50:56.064095 26483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Bootstrap starting.
I20260808 04:50:56.071607 26483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:50:56.071909 26483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Bootstrap complete.
I20260808 04:50:56.072597 26483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } }
I20260808 04:50:56.072738 26483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03ce5b0398a04b92b2311e6ab9f2aa5a, State: Initialized, Role: LEARNER
I20260808 04:50:56.072861 26483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } }
I20260808 04:50:56.073210 26490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } }
I20260808 04:50:56.073333 26483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:56.073411 26490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEARNER
I20260808 04:50:56.076665 26516 catalog_manager.cc:1259] Loaded cluster ID: eb89cc6dc32e401bbf5d3acafbe85a83
I20260808 04:50:56.076786 26516 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: loading cluster ID for follower catalog manager: success
I20260808 04:50:56.079078 26516 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: acquiring CA information for follower catalog manager: success
I20260808 04:50:56.080585 26516 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 04:50:56.116173 26374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:56.116490 26374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:56.116600 26374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:56.120671 26374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:50:56.120724 26374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:56.120743 26374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:56.120760 26374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:50:56.120810 26374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:50:56.126030 26374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.124.keytab
--principal=oryx/127.15.30.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.30.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:56.127065 26374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:56.128063 26374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:56.136719 26528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.136703 26526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.136812 26374 server_base.cc:1034] running on GCE node
W20260808 04:50:56.136706 26530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.137607 26374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:56.138713 26374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:56.139910 26374 hybrid_clock.cc:648] HybridClock initialized: now 1786164656139900 us; error 31 us; skew 500 ppm
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:56.142868 26374 init.cc:377] Logged in from keytab as oryx/127.15.30.124@KRBTEST.COM (short username oryx)
I20260808 04:50:56.143759 26374 webserver.cc:460] Webserver started at http://127.15.30.124:42219/ using document root <none> and password file <none>
I20260808 04:50:56.144353 26374 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:56.144500 26374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:56.144757 26374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:50:56.146880 26374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/instance:
uuid: "fabbd9170046444782edc6cadd9eacda"
format_stamp: "Formatted at 2026-08-08 04:50:56 on dist-test-slave-zpvx"
server_key: "809d24fe2ee965b516725d7d8f5cee13"
server_key_iv: "ce1ee615b30d53c7e9af5c7a55298bfa"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.147482 26374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal/instance:
uuid: "fabbd9170046444782edc6cadd9eacda"
format_stamp: "Formatted at 2026-08-08 04:50:56 on dist-test-slave-zpvx"
server_key: "809d24fe2ee965b516725d7d8f5cee13"
server_key_iv: "ce1ee615b30d53c7e9af5c7a55298bfa"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.151429 26374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 04:50:56.153990 26537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.155275 26374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:56.155426 26374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
uuid: "fabbd9170046444782edc6cadd9eacda"
format_stamp: "Formatted at 2026-08-08 04:50:56 on dist-test-slave-zpvx"
server_key: "809d24fe2ee965b516725d7d8f5cee13"
server_key_iv: "ce1ee615b30d53c7e9af5c7a55298bfa"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.155539 26374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:56.176769 26374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:56.179803 26374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:56.180035 26374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:56.192335 26374 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.124:40143
I20260808 04:50:56.192746 26615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.124:40143 every 8 connection(s)
I20260808 04:50:56.193656 26374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
I20260808 04:50:56.196712 26616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:56.197978 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 26374
I20260808 04:50:56.201203 26616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.30.124@KRBTEST.COM
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.124@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:50:56.214623 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.199650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.15.30.124:40143)
0808 04:50:56.200154 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.200208 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.200245 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:50:56.201314 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.201335 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.201756 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.202064 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.202970 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.204444 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.205628 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.206296 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.206481 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.209419 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.209425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.209434 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.209455 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.211406 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.211920 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.211926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.211931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.212049 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.212496 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.212501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.212505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.212978 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.213335 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.213498 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.213786 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":166,"threads_started":1}
I20260808 04:50:56.215078 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.203306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.15.30.126:34745)
0808 04:50:56.203439 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.203444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.203775 (+ 331us) server_negotiation.cc:407] Connection header received
0808 04:50:56.203940 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.203943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.203995 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.204085 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.205209 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.205874 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.207276 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.207494 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.207579 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.211008 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.211011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.211013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.211038 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.212692 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.213247 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.213250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.213252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.213351 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.213673 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.213677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.213678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.213898 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.213972 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.214457 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.214583 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:50:56.215839 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.203106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:36942)
0808 04:50:56.203652 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.203691 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.203833 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.204128 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.204134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.204758 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.205061 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.205071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.206139 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.206142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.207143 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.207157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.207664 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.207669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.207813 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.209009 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.209035 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.210857 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.212808 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.212816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.212821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.213110 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.213472 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.213474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.213476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.213562 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.214074 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.214081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.214224 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.214369 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.214536 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":218,"threads_started":1}
I20260808 04:50:56.217219 26616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.124@KRBTEST.COM for oryx/127.15.30.125@KRBTEST.COM
I20260808 04:50:56.225325 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.217751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.0.0.1:55494)
0808 04:50:56.217876 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.217887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.217945 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.218411 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.218414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.218567 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.218732 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.218738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.219652 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.219656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.220156 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.220163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.220670 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.220673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.220820 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.221177 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.221188 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.222446 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.224074 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.224077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.224078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.224261 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.224570 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.224572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.224573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.224669 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.224993 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.224995 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.225062 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.225139 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.225225 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:50:56.225466 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.217877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.15.30.125:34027)
0808 04:50:56.218193 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.218197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.218209 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:50:56.218245 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.218247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.218285 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.218399 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.218859 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.219532 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.220271 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.220549 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.220660 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.222585 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.222588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.222590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.222612 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.223972 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.224348 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.224351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.224352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.224473 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.224748 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.224750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.224751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.224915 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.224992 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.225199 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.225294 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":131,"threads_started":1}
I20260808 04:50:56.226122 26616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.124@KRBTEST.COM for oryx/127.15.30.124@KRBTEST.COM
I20260808 04:50:56.233318 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.226518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.0.0.1:59460)
0808 04:50:56.226625 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.226637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.226694 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.226923 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.226925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.227042 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.227104 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.227108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.228278 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.228281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.228815 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.228824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.229257 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.229260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.229313 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.229664 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.229674 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.230706 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.232102 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.232105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.232106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.232288 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.232581 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.232584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.232585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.232664 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.232997 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.232999 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.233064 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.233141 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.233223 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 04:50:56.233434 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.226662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.15.30.124:40143)
0808 04:50:56.226764 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.226767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.226778 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:50:56.226805 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.226807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.226837 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.226900 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.227206 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.227747 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.228932 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.229156 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.229246 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.230828 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.230831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.230833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.230853 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.231996 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.232409 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.232416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.232420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.232500 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.232760 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.232762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.232763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.232915 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.232996 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.233201 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.233286 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 04:50:56.235899 26616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Bootstrap starting.
I20260808 04:50:56.236090 25785 heartbeater.cc:503] Master 127.15.30.126:34745 was elected leader, sending a full tablet report...
I20260808 04:50:56.238483 26616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:56.239389 26616 log.cc:824] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:56.241247 26616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: No bootstrap required, opened a new log
I20260808 04:50:56.245245 26616 raft_consensus.cc:348] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } }
I20260808 04:50:56.245453 26616 raft_consensus.cc:374] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:56.245528 26616 raft_consensus.cc:729] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fabbd9170046444782edc6cadd9eacda, State: Initialized, Role: FOLLOWER
I20260808 04:50:56.245971 26616 consensus_queue.cc:260] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } }
I20260808 04:50:56.246512 26628 sys_catalog.cc:455] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } } }
I20260808 04:50:56.246644 26628 sys_catalog.cc:458] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:50:56.247341 26616 sys_catalog.cc:564] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:50:56.253249 26643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:50:56.253356 26643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:50:56.256142 26374 master_runner.cc:428] Detected that this master fabbd9170046444782edc6cadd9eacda is joining an existing cluster
I20260808 04:50:56.256196 26374 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.30.124:40143
I20260808 04:50:56.268035 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 25568
I20260808 04:50:56.271780 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.258870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.15.30.124:40143)
0808 04:50:56.258975 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.258980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.259529 (+ 549us) server_negotiation.cc:407] Connection header received
0808 04:50:56.259706 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.259709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.259745 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.259833 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.260872 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.261616 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.263731 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.264061 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.264125 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.265773 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.265776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.265779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.265803 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.267400 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.268039 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.268043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.268045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.268193 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.270887 (+ 2694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.270892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.270896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.271088 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.271159 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.271445 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.271554 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":32}
I20260808 04:50:56.272361 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.258975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.0.0.1:59466)
0808 04:50:56.259421 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.259444 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.259594 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.259850 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.259852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.260347 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.260748 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.260760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.261741 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.261744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.263604 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.263616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.264286 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.264290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.264551 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.265185 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.265201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.265613 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.267516 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.267519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.267522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.267838 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.270696 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.270700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.270702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.270800 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.271211 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.271215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.271310 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.271995 (+ 685us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.272132 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":189,"threads_started":1}
I20260808 04:50:56.293146 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.258397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.15.30.126:34745)
0808 04:50:56.258523 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.258528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.258944 (+ 416us) server_negotiation.cc:407] Connection header received
0808 04:50:56.259173 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.259176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.259232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.259316 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.260742 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.261621 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.263190 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.263544 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.263625 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.281002 (+ 17377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.281006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.281009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.281040 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.290396 (+ 9356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.290940 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.290946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.290950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.291041 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.291706 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.291712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.291717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.291991 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.292071 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.292506 (+ 435us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.292652 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:50:56.293817 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.258381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:36952)
0808 04:50:56.258862 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.258877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.259016 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.259455 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.259461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.259771 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.260594 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.260606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.261805 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.261809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.263033 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.263045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.263802 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.263806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.263946 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.264620 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.264636 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.280698 (+ 16062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.290520 (+ 9822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.290526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.290531 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.290813 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.291158 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.291163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.291168 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.291278 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.292150 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.292156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.292280 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.293340 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.293534 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":133,"threads_started":1}
I20260808 04:50:56.295732 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.258536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.15.30.125:34027)
0808 04:50:56.258654 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.258660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.260569 (+ 1909us) server_negotiation.cc:407] Connection header received
0808 04:50:56.261100 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.261103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.261148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.261196 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.262171 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.263030 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.265275 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.265644 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.265750 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.291051 (+ 25301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.291055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.291058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.291086 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.293605 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.294433 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.294437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.294438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.294534 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.294871 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.294875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.294876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.295065 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.295171 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.295444 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.295578 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 04:50:56.295696 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.259575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.0.0.1:55500)
0808 04:50:56.260411 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.260437 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.260970 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.261327 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.261330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.261655 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.261938 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.261946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.263227 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.263232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.264727 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.264742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.265888 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.265892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.265980 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.289566 (+ 23586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.289581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.290880 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.293848 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.293851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.293853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.294310 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.294647 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.294650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.294652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.294742 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.295172 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.295176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.295273 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.295371 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.295515 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":1,"thread_start_us":581,"threads_started":1}
I20260808 04:50:56.298898 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:40717
--local_ip_for_outbound_sockets=127.15.30.65
--tserver_master_addrs=127.15.30.126:34745
--webserver_port=46455
--webserver_interface=127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab
--principal=oryx/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164654, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.126@KRBTEST.COM for oryx/127.15.30.124@KRBTEST.COM
I20260808 04:50:56.306231 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.298063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.15.30.124:40143)
0808 04:50:56.298196 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.298202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.298235 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:50:56.298304 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.298306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.298346 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.298431 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.299015 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.299766 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.300920 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.301159 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.301271 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.303128 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.303132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.303134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.303156 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.304652 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.305062 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.305066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.305068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.305142 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.305422 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.305426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.305427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.305617 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.305738 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.305996 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.306099 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:50:56.306233 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.297724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.0.0.1:59476)
0808 04:50:56.297851 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.297867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.297948 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.298559 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.298562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.298760 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.298879 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.298887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.299905 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.299911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.300801 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.300811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.301284 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.301288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.301385 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.301811 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.301831 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.303004 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.304766 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.304771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.304773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.304966 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.305237 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.305241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.305243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.305327 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.305733 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.305736 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.305832 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.305932 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.306068 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 04:50:56.307418 25464 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.30.124:40143: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fabbd9170046444782edc6cadd9eacda"
member_type: NON_VOTER
last_known_addr {
host: "127.15.30.124"
port: 40143
}
attrs {
promote: true
}
}
dest_uuid: "f5a1c13a8d4244699977da3aceb3a1da"
cas_config_opid_index: 5
I20260808 04:50:56.307973 25464 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } }
I20260808 04:50:56.309091 26342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fabbd9170046444782edc6cadd9eacda (127.15.30.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.310731 26342 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.30.124:40143
I20260808 04:50:56.310892 26342 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.310978 26342 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:56.311913 26572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:56.313367 26572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Refusing update from remote peer f5a1c13a8d4244699977da3aceb3a1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:50:56.313970 26374 master.cc:537] Master@127.15.30.124:40143 shutting down...
I20260808 04:50:56.314337 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.309442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.15.30.125:34027)
0808 04:50:56.309636 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.309641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.309875 (+ 234us) server_negotiation.cc:407] Connection header received
0808 04:50:56.309917 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.309920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.309970 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.310048 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:56.310401 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.311127 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.312750 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.313393 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.313571 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.313724 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.314107 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.314209 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260808 04:50:56.314872 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.309219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.0.0.1:55502)
0808 04:50:56.309717 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.309730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.309803 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.310158 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.310161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.310177 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:56.310275 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.310281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.311350 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.311354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.312602 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.312614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.313796 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.313799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.313889 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.313936 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.314013 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.314765 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435}
I20260808 04:50:56.315693 26436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Refusing update from remote peer f5a1c13a8d4244699977da3aceb3a1da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:50:56.316237 26344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 04:50:56.316577 26344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:50:56.317757 25441 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 04:50:56.317844 26436 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fabbd9170046444782edc6cadd9eacda (127.15.30.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
W20260808 04:50:56.318713 25441 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da -> Peer fabbd9170046444782edc6cadd9eacda (127.15.30.124:40143): Couldn't send request to peer fabbd9170046444782edc6cadd9eacda. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 04:50:56.319446 26658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a1c13a8d4244699977da3aceb3a1da. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.319424 26490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: NON_VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.319715 26490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEARNER
I20260808 04:50:56.319864 26658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEARNER
I20260808 04:50:56.322904 26342 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: attempting to promote NON_VOTER 03ce5b0398a04b92b2311e6ab9f2aa5a to VOTER
I20260808 04:50:56.323305 26342 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } }
I20260808 04:50:56.324749 26436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 LEARNER]: Refusing update from remote peer f5a1c13a8d4244699977da3aceb3a1da: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 04:50:56.325146 26374 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:56.325172 26344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 04:50:56.325416 25441 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da -> Peer fabbd9170046444782edc6cadd9eacda (127.15.30.124:40143): Couldn't send request to peer fabbd9170046444782edc6cadd9eacda. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 04:50:56.325426 26374 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:56.325527 26374 tablet_replica.cc:330] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: stopping tablet replica
I20260808 04:50:56.328075 26659 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 03ce5b0398a04b92b2311e6ab9f2aa5a (127.15.30.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.328807 26436 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 03ce5b0398a04b92b2311e6ab9f2aa5a (127.15.30.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.329857 26659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.330039 26659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:56.330231 26659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.330390 26659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:56.331374 26658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.331579 26658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:50:56.332985 26374 master.cc:552] Master@127.15.30.124:40143 shutdown complete.
I20260808 04:50:56.333050 26374 master_runner.cc:305] Clearing existing system tablet
I20260808 04:50:56.333271 26374 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:56.333336 26374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:56.334379 26374 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 04:50:56.335209 26663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.335404 26374 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:56.335574 26374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:56.335613 26374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
uuid: "fabbd9170046444782edc6cadd9eacda"
format_stamp: "Formatted at 2026-08-08 04:50:56 on dist-test-slave-zpvx"
server_key: "809d24fe2ee965b516725d7d8f5cee13"
server_key_iv: "ce1ee615b30d53c7e9af5c7a55298bfa"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.337255 26374 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:56.338986 26374 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:50:56.339088 26374 log.cc:1196] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 04:50:56.339321 26374 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Deleting consensus metadata
I20260808 04:50:56.339501 26374 master_runner.cc:315] Copying system tablet from 127.15.30.126:34745
I20260808 04:50:56.340579 26374 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: tablet copy: Beginning tablet copy session from remote peer at address 127.15.30.126:34745
I20260808 04:50:56.348146 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.341049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:36962)
0808 04:50:56.341343 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.341355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.341468 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.341802 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.341804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.341967 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.342043 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.342048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.342797 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.342801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.343394 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.343401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.343855 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.343858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.343938 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.344509 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.344528 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.344851 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.346571 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.346577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.346579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.346792 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.347066 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.347070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.347072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.347159 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.347559 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.347563 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.347688 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.347804 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.347943 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":98,"threads_started":1}
I20260808 04:50:56.348142 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.341091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.15.30.126:34745)
0808 04:50:56.341204 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.341208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.341426 (+ 218us) server_negotiation.cc:407] Connection header received
0808 04:50:56.341552 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.341555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.341595 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.341685 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.342162 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.342690 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.343513 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.343747 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.343850 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.344995 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.344998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.345001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.345026 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.346458 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.346890 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.346894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.346896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.346977 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.347249 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.347253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.347255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.347466 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.347570 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.347863 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.347983 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:50:56.348904 25495 tablet_copy_service.cc:140] P f5a1c13a8d4244699977da3aceb3a1da: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fabbd9170046444782edc6cadd9eacda ({username='oryx', principal='oryx/127.15.30.124@KRBTEST.COM'} at 127.0.0.1:36962)
I20260808 04:50:56.349009 25495 tablet_copy_service.cc:161] P f5a1c13a8d4244699977da3aceb3a1da: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fabbd9170046444782edc6cadd9eacda at {username='oryx', principal='oryx/127.15.30.124@KRBTEST.COM'} at 127.0.0.1:36962: session id = fabbd9170046444782edc6cadd9eacda-00000000000000000000000000000000
I20260808 04:50:56.350090 25495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:50:56.351161 26374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:50:56.352957 26374 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: tablet copy: Starting download of 0 data blocks...
I20260808 04:50:56.353149 26374 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: tablet copy: Starting download of 1 WAL segments...
I20260808 04:50:56.354455 26374 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:50:56.355873 25495 tablet_copy_service.cc:342] P f5a1c13a8d4244699977da3aceb3a1da: Request end of tablet copy session fabbd9170046444782edc6cadd9eacda-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.30.124@KRBTEST.COM'} at 127.0.0.1:36962
I20260808 04:50:56.355954 25495 tablet_copy_service.cc:433] P f5a1c13a8d4244699977da3aceb3a1da: ending tablet copy session fabbd9170046444782edc6cadd9eacda-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fabbd9170046444782edc6cadd9eacda
I20260808 04:50:56.362346 26374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:56.363930 26674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.363986 26677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.364253 26675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.364283 26374 server_base.cc:1034] running on GCE node
I20260808 04:50:56.364575 26374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:56.364881 26374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:56.366025 26374 hybrid_clock.cc:648] HybridClock initialized: now 1786164656366034 us; error 41 us; skew 500 ppm
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:56.367496 26374 init.cc:377] Logged in from keytab as oryx/127.15.30.124@KRBTEST.COM (short username oryx)
I20260808 04:50:56.367990 26374 webserver.cc:460] Webserver started at http://127.15.30.124:35791/ using document root <none> and password file <none>
I20260808 04:50:56.368180 26374 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:56.368224 26374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:56.369500 26374 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260808 04:50:56.370390 26695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.370613 26374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:56.370710 26374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
uuid: "fabbd9170046444782edc6cadd9eacda"
format_stamp: "Formatted at 2026-08-08 04:50:56 on dist-test-slave-zpvx"
server_key: "809d24fe2ee965b516725d7d8f5cee13"
server_key_iv: "ce1ee615b30d53c7e9af5c7a55298bfa"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.370795 26374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:56.391186 26374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:56.393426 26374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:56.398468 26374 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.124:40143
I20260808 04:50:56.399104 26374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
I20260808 04:50:56.399524 26768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.124:40143 every 8 connection(s)
I20260808 04:50:56.400699 26771 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:50:56.401503 26771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Bootstrap starting.
I20260808 04:50:56.410418 26771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:50:56.410713 26771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: Bootstrap complete.
I20260808 04:50:56.411208 26771 raft_consensus.cc:348] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } }
I20260808 04:50:56.411347 26771 raft_consensus.cc:729] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fabbd9170046444782edc6cadd9eacda, State: Initialized, Role: LEARNER
I20260808 04:50:56.411469 26771 consensus_queue.cc:260] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } }
I20260808 04:50:56.411803 26775 sys_catalog.cc:455] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.411960 26771 sys_catalog.cc:564] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:50:56.411986 26775 sys_catalog.cc:458] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: This master's current role is: LEARNER
I20260808 04:50:56.415781 26788 catalog_manager.cc:1259] Loaded cluster ID: eb89cc6dc32e401bbf5d3acafbe85a83
I20260808 04:50:56.415843 26788 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: loading cluster ID for follower catalog manager: success
I20260808 04:50:56.417528 26788 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: acquiring CA information for follower catalog manager: success
I20260808 04:50:56.418603 26788 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:50:56.444756 25992 heartbeater.cc:503] Master 127.15.30.126:34745 was elected leader, sending a full tablet report...
W20260808 04:50:56.463529 26654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:56.463768 26654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:56.463836 26654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:56.468233 26654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:56.468293 26654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:56.468398 26654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:50:56.474255 26654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab
--principal=oryx/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:40717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=46455
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:56.475262 26654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:56.476281 26654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:56.486564 26794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.486677 26795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.486570 26798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.486629 26654 server_base.cc:1034] running on GCE node
I20260808 04:50:56.487186 26654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:56.487779 26654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:56.488940 26654 hybrid_clock.cc:648] HybridClock initialized: now 1786164656488924 us; error 27 us; skew 500 ppm
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:56.491948 26654 init.cc:377] Logged in from keytab as oryx/127.15.30.65@KRBTEST.COM (short username oryx)
I20260808 04:50:56.492833 26654 webserver.cc:460] Webserver started at http://127.15.30.65:46455/ using document root <none> and password file <none>
I20260808 04:50:56.493382 26654 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:56.493492 26654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:56.497434 26654 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:50:56.499794 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.501194 26654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:50:56.501327 26654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "2eaf5c3842aa4ad28cfae11225fd345c"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "5c120f0508e2d074344842bf5d981cfe"
server_key_iv: "ed2069aa9765e6277be0b19f1fca769d"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.501914 26654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:56.515731 26654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:56.519125 26654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:56.519367 26654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:56.520049 26654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:56.521037 26654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:56.521107 26654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.521162 26654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:56.521196 26654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.538573 26654 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:40717
I20260808 04:50:56.538602 26998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:40717 every 8 connection(s)
I20260808 04:50:56.539556 26654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:50:56.544631 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 26654
I20260808 04:50:56.544829 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 25791
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.124@KRBTEST.COM
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.125@KRBTEST.COM
I20260808 04:50:56.559355 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:40251
--local_ip_for_outbound_sockets=127.15.30.66
--tserver_master_addrs=127.15.30.126:34745
--webserver_port=32787
--webserver_interface=127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab
--principal=oryx/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:50:56.559804 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.541873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:51409 (local address 127.15.30.125:34027)
0808 04:50:56.541988 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.541993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.543902 (+ 1909us) server_negotiation.cc:407] Connection header received
0808 04:50:56.544446 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.544450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.544509 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.544569 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.546821 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.547823 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.550157 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.550562 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.550655 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.555644 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.555649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.555653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.555685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.557437 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.557960 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.557966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.557971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.558085 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.558445 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.558452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.558456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.558705 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.558793 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.559419 (+ 626us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.559572 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:50:56.559814 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.541869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:45057 (local address 127.15.30.124:40143)
0808 04:50:56.542323 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.542333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.543527 (+ 1194us) server_negotiation.cc:407] Connection header received
0808 04:50:56.544493 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.544498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.544564 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.544637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.546767 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.548039 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.550214 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.550687 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.550779 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.555200 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.555206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.555213 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.555249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.557092 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.557766 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.557770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.557773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.557871 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.558325 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.558330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.558331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.558589 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.558688 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.559428 (+ 740us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.559582 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":118,"threads_started":1}
I20260808 04:50:56.561579 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.542151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.15.30.65:51409)
0808 04:50:56.543766 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.543781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.544179 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.545006 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.545017 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.545736 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.546626 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.546643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.548051 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.548057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.550002 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.550016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.550876 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.550882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.551235 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.552051 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.552080 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.555445 (+ 3365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.557567 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.557573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.557578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.557842 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.558216 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.558222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.558227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.558332 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.558858 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.558866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.559119 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.560977 (+ 1858us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.561176 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1516,"mutex_wait_us":462,"spinlock_wait_cycles":39808,"thread_start_us":211,"threads_started":1}
I20260808 04:50:56.561640 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.542151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.15.30.65:45057)
0808 04:50:56.543416 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.543432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.544190 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.545029 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.545034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.545732 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.546551 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.546573 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.548206 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.548210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.550054 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.550068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.550950 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.550956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.551242 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.552013 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.552047 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.554938 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.557227 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.557232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.557247 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.557624 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.558003 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.558008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.558011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.558200 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.558762 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.558768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.559120 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.559985 (+ 865us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.560356 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":248,"spinlock_wait_cycles":36608,"thread_start_us":278,"threads_started":1}
I20260808 04:50:56.563383 26999 heartbeater.cc:347] Connected to a master server at 127.15.30.124:40143
I20260808 04:50:56.563645 26999 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:56.564245 26999 heartbeater.cc:511] Master 127.15.30.124:40143 requested a full tablet report, sending...
I20260808 04:50:56.565169 27001 heartbeater.cc:347] Connected to a master server at 127.15.30.125:34027
I20260808 04:50:56.565264 27001 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:56.565441 27001 heartbeater.cc:511] Master 127.15.30.125:34027 requested a full tablet report, sending...
I20260808 04:50:56.566251 26725 ts_manager.cc:194] Registered new tserver with Master: 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717)
I20260808 04:50:56.566566 26422 ts_manager.cc:194] Registered new tserver with Master: 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717)
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:50:56.573415 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.542150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.65:48341)
0808 04:50:56.542953 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.543088 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.544179 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.561548 (+ 17369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.561551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.561774 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.561876 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.561882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.563108 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.563110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.563945 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.563954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.566083 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.566087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.566172 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.566833 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.566853 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.569104 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.571444 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.571449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.571454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.571745 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.572087 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.572091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.572095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.572195 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.572751 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.572756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.572860 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.572981 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.573130 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":307,"threads_started":1}
I20260808 04:50:56.573778 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.541508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:48341 (local address 127.15.30.126:34745)
0808 04:50:56.541664 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.541669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.543186 (+ 1517us) server_negotiation.cc:407] Connection header received
0808 04:50:56.561262 (+ 18076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.561282 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.561344 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.561406 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.562036 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.562786 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.564058 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.564518 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.564642 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.569258 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.569264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.569269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.569307 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.571320 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.571862 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.571867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.571870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.571975 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.572306 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.572313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.572316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.572625 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.572759 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.573072 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.573245 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:50:56.574671 27000 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34745
I20260808 04:50:56.574754 27000 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:56.574877 27000 heartbeater.cc:511] Master 127.15.30.126:34745 requested a full tablet report, sending...
I20260808 04:50:56.575757 25464 ts_manager.cc:194] Re-registered known tserver with Master: 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717)
I20260808 04:50:56.576756 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:48341
I20260808 04:50:56.655912 26192 heartbeater.cc:503] Master 127.15.30.126:34745 was elected leader, sending a full tablet report...
W20260808 04:50:56.687979 27008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:56.688320 27008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:56.688409 27008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:56.694008 27008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:56.694080 27008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:56.694170 27008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:50:56.699342 27008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab
--principal=oryx/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:40251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=32787
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:56.700280 27008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:56.701254 27008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:56.711015 27014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.711030 27017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.711151 27015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.711447 27008 server_base.cc:1034] running on GCE node
I20260808 04:50:56.711839 27008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:56.712433 27008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:56.713629 27008 hybrid_clock.cc:648] HybridClock initialized: now 1786164656713609 us; error 34 us; skew 500 ppm
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:56.716689 27008 init.cc:377] Logged in from keytab as oryx/127.15.30.66@KRBTEST.COM (short username oryx)
I20260808 04:50:56.717455 27008 webserver.cc:460] Webserver started at http://127.15.30.66:32787/ using document root <none> and password file <none>
I20260808 04:50:56.717948 27008 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:56.718020 27008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:56.721707 27008 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:50:56.723630 27028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.724861 27008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:56.725010 27008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "88b1338766d540a584b8fd50b4e950b9"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "bd79ab533dc3e9592f98109aef1fbee8"
server_key_iv: "e6eff8657afb182790faeaaf6dda1366"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.725508 27008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:56.751235 27008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:56.754861 27008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:56.755096 27008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:56.755699 27008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:56.756664 27008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:56.756734 27008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.756804 27008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:56.756855 27008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.767198 27008 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:40251
I20260808 04:50:56.767236 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:40251 every 8 connection(s)
I20260808 04:50:56.768338 27008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:50:56.769013 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 27008
I20260808 04:50:56.769241 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 26000
I20260808 04:50:56.782465 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:46127
--local_ip_for_outbound_sockets=127.15.30.67
--tserver_master_addrs=127.15.30.126:34745
--webserver_port=36067
--webserver_interface=127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab
--principal=oryx/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.125@KRBTEST.COM
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.124@KRBTEST.COM
I20260808 04:50:56.788615 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.771725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:60839 (local address 127.15.30.124:40143)
0808 04:50:56.771970 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.771978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.775684 (+ 3706us) server_negotiation.cc:407] Connection header received
0808 04:50:56.775866 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.775869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.775922 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.775975 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.776616 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.777429 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.779591 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.779939 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.780007 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.784509 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.784512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.784515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.784541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.786550 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.787125 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.787131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.787136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.787223 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.787541 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.787547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.787551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.787756 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.787846 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.788290 (+ 444us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.788443 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:50:56.788615 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.771561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:47353 (local address 127.15.30.126:34745)
0808 04:50:56.771823 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.771829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.774731 (+ 2902us) server_negotiation.cc:407] Connection header received
0808 04:50:56.774909 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.774915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.774974 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.775052 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.776608 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.777430 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.779582 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.779925 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.780006 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.784509 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.784512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.784515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.784541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.786634 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.787203 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.787206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.787208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.787294 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.787593 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.787598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.787600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.787797 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.787877 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.788291 (+ 414us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.788444 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260808 04:50:56.788743 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.771660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:52959 (local address 127.15.30.125:34027)
0808 04:50:56.771936 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.771941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.773389 (+ 1448us) server_negotiation.cc:407] Connection header received
0808 04:50:56.774457 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.774461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.774524 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.774616 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.776608 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.777499 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.779596 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.780003 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.780098 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.783608 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.783615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.783621 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.783654 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.785849 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.786520 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.786526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.786531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.786639 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.787146 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.787153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.787158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.787432 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.787564 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.788317 (+ 753us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.788499 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260808 04:50:56.790352 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.771263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.66:47353)
0808 04:50:56.774652 (+ 3389us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.774665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.774797 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.775202 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.775208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.775634 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.776426 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.776449 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.777623 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.777626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.779434 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.779445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.780213 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.780217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.780569 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.781260 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.781283 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.784319 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.786757 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.786762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.786766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.787096 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.787398 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.787401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.787403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.787477 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.787926 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.787928 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.788031 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.788825 (+ 794us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.789263 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3299,"mutex_wait_us":1192,"spinlock_wait_cycles":5376,"thread_start_us":132,"threads_started":1}
I20260808 04:50:56.790611 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.771264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.15.30.66:60839)
0808 04:50:56.775593 (+ 4329us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.775609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.775755 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.776113 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.776116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.776297 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.776440 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.776449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.777623 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.777626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.779434 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.779445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.780213 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.780217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.780682 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.781531 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.781551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.784319 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.786682 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.786685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.786687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.786989 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.787336 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.787342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.787346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.787433 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.787911 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.787915 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.788032 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.788897 (+ 865us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.789313 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4221,"mutex_wait_us":1720,"spinlock_wait_cycles":10112,"thread_start_us":145,"threads_started":1}
I20260808 04:50:56.790354 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.771263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.15.30.66:52959)
0808 04:50:56.773232 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.773289 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.774232 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.774907 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.774943 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.775686 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.776429 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.776452 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.777651 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.777658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.779452 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.779467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.780279 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.780285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.780673 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.781406 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.781427 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.783374 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.785990 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.785999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.786016 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.786394 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.786759 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.786765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.786770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.787014 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.787650 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.787659 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.787985 (+ 326us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.789462 (+ 1477us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.789655 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"mutex_wait_us":324,"spinlock_wait_cycles":14720,"thread_start_us":740,"threads_started":1}
I20260808 04:50:56.791610 27199 heartbeater.cc:347] Connected to a master server at 127.15.30.124:40143
I20260808 04:50:56.791643 27201 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34745
I20260808 04:50:56.791739 27202 heartbeater.cc:347] Connected to a master server at 127.15.30.125:34027
I20260808 04:50:56.791849 27199 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:56.791846 27201 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:56.791846 27202 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:56.792497 27202 heartbeater.cc:511] Master 127.15.30.125:34027 requested a full tablet report, sending...
I20260808 04:50:56.792497 27201 heartbeater.cc:511] Master 127.15.30.126:34745 requested a full tablet report, sending...
I20260808 04:50:56.792546 27199 heartbeater.cc:511] Master 127.15.30.124:40143 requested a full tablet report, sending...
I20260808 04:50:56.793740 26422 ts_manager.cc:194] Registered new tserver with Master: 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251)
I20260808 04:50:56.793932 26725 ts_manager.cc:194] Registered new tserver with Master: 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251)
I20260808 04:50:56.794147 25464 ts_manager.cc:194] Re-registered known tserver with Master: 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251)
I20260808 04:50:56.794780 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:47353
I20260808 04:50:56.889698 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.885971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.15.30.124:40143)
0808 04:50:56.886134 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.886141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.886259 (+ 118us) server_negotiation.cc:407] Connection header received
0808 04:50:56.886374 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.886377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.886448 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.886561 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:56.886787 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.887578 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.888677 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.889141 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.889257 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.889415 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:56.889494 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 04:50:56.889562 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:50:56.889703 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.885831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.0.0.1:59484)
0808 04:50:56.886169 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.886182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.886286 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.886567 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.886569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.886583 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:56.886678 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.886683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.887682 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.887684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.888545 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.888557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.889274 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.889278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.889368 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.889392 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.889468 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.889550 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":80,"threads_started":1}
I20260808 04:50:56.890291 26739 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 04:50:56.890643 26775 sys_catalog.cc:455] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a1c13a8d4244699977da3aceb3a1da. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:56.890793 26775 sys_catalog.cc:458] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: This master's current role is: LEARNER
W20260808 04:50:56.898963 27208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:50:56.899214 27208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:50:56.899312 27208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:50:56.903180 27208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:50:56.903254 27208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:50:56.903345 27208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:50:56.908967 27208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab
--principal=oryx/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:46127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=36067
--tserver_master_addrs=127.15.30.126:34745,127.15.30.125:34027,127.15.30.124:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:50:56.909950 27208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:50:56.910943 27208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:50:56.920665 27220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.920753 27208 server_base.cc:1034] running on GCE node
W20260808 04:50:56.920670 27222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:50:56.920713 27218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:50:56.921265 27208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:50:56.921908 27208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:50:56.923113 27208 hybrid_clock.cc:648] HybridClock initialized: now 1786164656923078 us; error 30 us; skew 500 ppm
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:50:56.925854 27208 init.cc:377] Logged in from keytab as oryx/127.15.30.67@KRBTEST.COM (short username oryx)
I20260808 04:50:56.926604 27208 webserver.cc:460] Webserver started at http://127.15.30.67:36067/ using document root <none> and password file <none>
I20260808 04:50:56.927099 27208 fs_manager.cc:359] Metadata directory not provided
I20260808 04:50:56.927173 27208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:50:56.930938 27208 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:56.932862 27229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.934093 27208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:56.934199 27208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "cd1a534a60b44186a8a079e9d4c4c0cc"
format_stamp: "Formatted at 2026-08-08 04:50:55 on dist-test-slave-zpvx"
server_key: "907459356c98fd4b58fe33c7dcc22c58"
server_key_iv: "4ff0f88e2a038fb11a7d6ddcfa71550f"
server_key_version: "encryptionkey@0"
I20260808 04:50:56.934679 27208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:50:56.966050 27208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:50:56.969403 27208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:50:56.969627 27208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:50:56.970193 27208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:50:56.971108 27208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:50:56.971174 27208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.971246 27208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:50:56.971284 27208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:50:56.980499 27208 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:46127
I20260808 04:50:56.980523 27403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:46127 every 8 connection(s)
I20260808 04:50:56.981463 27208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:50:56.990051 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 27208
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.125@KRBTEST.COM
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
Aug 08 04:50:56 dist-test-slave-zpvx krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164656, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.124@KRBTEST.COM
I20260808 04:50:57.000903 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.984082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.15.30.67:33123)
0808 04:50:56.984791 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.984843 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.985895 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.986483 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.986486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.986990 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.987565 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.987582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.988709 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.988713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.989650 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.989659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.990391 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.990687 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.991505 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.991527 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.995087 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.997431 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.997437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.997452 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.997851 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.998182 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.998185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.998187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.998389 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:56.998971 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:56.998978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:56.999274 (+ 296us) client_negotiation.cc:769] Sending connection context
0808 04:50:56.999547 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 04:50:56.999876 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":94,"spinlock_wait_cycles":27392,"thread_start_us":262,"threads_started":1}
I20260808 04:50:57.001855 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.983278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:33123 (local address 127.15.30.125:34027)
0808 04:50:56.983534 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.983539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.984970 (+ 1431us) server_negotiation.cc:407] Connection header received
0808 04:50:56.986229 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.986248 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.986311 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.986395 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.987737 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.988581 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.989773 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.990112 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.995245 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.995249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.995251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.995282 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.997323 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.997959 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.997966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.997971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.998072 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.998518 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.998521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.998523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:56.998776 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:56.998874 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.001552 (+ 2678us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.001699 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260808 04:50:57.002931 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.983376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:43473 (local address 127.15.30.126:34745)
0808 04:50:56.983500 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.983505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.985649 (+ 2144us) server_negotiation.cc:407] Connection header received
0808 04:50:56.986106 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.986112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.986172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.986268 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.987719 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.988260 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990183 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.990516 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990586 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.995994 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.995998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.996001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.996031 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.998042 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:56.999756 (+ 1714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:56.999759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:56.999762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.000156 (+ 394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.000556 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.000560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.000564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.000825 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.000942 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.002008 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.002779 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:50:57.003332 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.984190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:43473)
0808 04:50:56.985513 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.985523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.985895 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.986466 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.986478 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.986990 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.987564 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.987579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.988536 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.988539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.990052 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.990063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.990748 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.990997 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.991777 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.991798 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.995822 (+ 4024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:56.999361 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:56.999364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:56.999366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:56.999619 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:57.000285 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:57.000291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:57.000293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:57.000415 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:57.001227 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:57.001230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:57.001322 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 04:50:57.001425 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:50:57.001601 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":250,"spinlock_wait_cycles":32896,"thread_start_us":286,"threads_started":1}
I20260808 04:50:57.003727 27410 heartbeater.cc:347] Connected to a master server at 127.15.30.125:34027
I20260808 04:50:57.003978 27410 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:57.004076 27408 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34745
I20260808 04:50:57.004174 27408 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:57.004639 27408 heartbeater.cc:511] Master 127.15.30.126:34745 requested a full tablet report, sending...
I20260808 04:50:57.005868 25464 ts_manager.cc:194] Re-registered known tserver with Master: cd1a534a60b44186a8a079e9d4c4c0cc (127.15.30.67:46127)
I20260808 04:50:57.006702 25464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:43473
I20260808 04:50:57.008255 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.993608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.15.30.125:34027)
0808 04:50:56.994897 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.994902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.994916 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:50:56.995078 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.995081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.995131 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.995215 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.996603 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.997586 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.998896 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.999284 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.999393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.001257 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:57.001260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:57.001263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:57.001287 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:57.005486 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.006523 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.006527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.006529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.006620 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.007007 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.007010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.007012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.007259 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.007386 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.007953 (+ 567us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.008080 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":88,"threads_started":1}
I20260808 04:50:57.009227 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.983259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:45189 (local address 127.15.30.124:40143)
0808 04:50:56.983407 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.983412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.985297 (+ 1885us) server_negotiation.cc:407] Connection header received
0808 04:50:56.986145 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.986149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.986208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.986268 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.987744 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.988567 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990167 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.990509 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990586 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.995114 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:56.995119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:56.995122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:56.995165 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:56.999254 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.007021 (+ 7767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.007025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.007026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.007107 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.007967 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.007970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.007972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.008185 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.008260 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.008925 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.009058 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:50:57.009791 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.984082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.15.30.67:45189)
0808 04:50:56.985204 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:56.985216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:56.985893 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:56.986505 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:56.986508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:56.986998 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:50:56.987568 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.987582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.988685 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.988687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.990003 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:56.990015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:56.990748 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:56.990751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:56.991687 (+ 936us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:56.992312 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:50:56.992327 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:50:56.994930 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:50:57.005655 (+ 10725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:57.005660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:57.005662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:57.006017 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:57.007751 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:50:57.007757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:50:57.007761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:50:57.007850 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:50:57.008347 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:50:57.008350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:50:57.008481 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 04:50:57.009376 (+ 895us) client_negotiation.cc:240] Negotiation successful
0808 04:50:57.009529 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":245,"spinlock_wait_cycles":48640,"thread_start_us":298,"threads_started":1}
I20260808 04:50:57.004642 27410 heartbeater.cc:511] Master 127.15.30.125:34027 requested a full tablet report, sending...
I20260808 04:50:57.010468 27404 heartbeater.cc:347] Connected to a master server at 127.15.30.124:40143
I20260808 04:50:57.010560 27404 heartbeater.cc:464] Registering TS with master...
I20260808 04:50:57.010696 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.994594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.15.30.126:34745)
0808 04:50:56.995396 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.995401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.997685 (+ 2284us) server_negotiation.cc:407] Connection header received
0808 04:50:56.997872 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.997878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.997934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:56.997994 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:56.999330 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.000118 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.001247 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.001558 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.001636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.003748 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:57.003753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:57.003756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:57.003785 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:57.006644 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.007508 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.007511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.007513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.007603 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.008811 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.008815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.008816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.009037 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.009159 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.010206 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.010400 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":78,"threads_started":1}
I20260808 04:50:57.010759 27404 heartbeater.cc:511] Master 127.15.30.124:40143 requested a full tablet report, sending...
I20260808 04:50:57.010953 26422 ts_manager.cc:194] Registered new tserver with Master: cd1a534a60b44186a8a079e9d4c4c0cc (127.15.30.67:46127)
I20260808 04:50:57.011427 26725 ts_manager.cc:194] Registered new tserver with Master: cd1a534a60b44186a8a079e9d4c4c0cc (127.15.30.67:46127)
I20260808 04:50:57.011644 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:56.993579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.15.30.124:40143)
0808 04:50:56.996451 (+ 2872us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:56.996456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:56.996469 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:50:56.996507 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:56.996509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:56.996557 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.002641 (+ 6084us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:57.002768 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.003560 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.004747 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.005065 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.005584 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.006812 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:57.006816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:57.006818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:57.006841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:57.008831 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.009355 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.009358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.009360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.009447 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.009785 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.009788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.009790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.010019 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.010720 (+ 701us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.010794 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.010905 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2751,"thread_start_us":167,"threads_started":1}
I20260808 04:50:57.013762 25464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:57.016922 25464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:57.019398 27433 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: NON_VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: true } } }
I20260808 04:50:57.019574 27433 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:57.023114 27430 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: attempting to promote NON_VOTER fabbd9170046444782edc6cadd9eacda to VOTER
I20260808 04:50:57.023474 27430 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } }
I20260808 04:50:57.024538 26436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Refusing update from remote peer f5a1c13a8d4244699977da3aceb3a1da: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 04:50:57.025041 26738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 LEARNER]: Refusing update from remote peer f5a1c13a8d4244699977da3aceb3a1da: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 04:50:57.025136 27209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 04:50:57.025430 27209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 04:50:57.026908 27433 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fabbd9170046444782edc6cadd9eacda (127.15.30.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.027380 26436 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fabbd9170046444782edc6cadd9eacda (127.15.30.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.027778 26738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fabbd9170046444782edc6cadd9eacda (127.15.30.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.028635 27433 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.028764 27433 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:57.028899 27433 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.028997 27433 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:57.029083 27209 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.029290 27209 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:57.029822 27449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.029956 27449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:50:57.030725 26775 sys_catalog.cc:455] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5a1c13a8d4244699977da3aceb3a1da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:57.030835 26775 sys_catalog.cc:458] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:50:57.037003 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.030789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:46127 (local address 127.0.0.1:59314)
0808 04:50:57.030914 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:57.030928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:57.031008 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:57.032855 (+ 1847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:57.032858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:57.032877 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:57.032993 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.032999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.034673 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.034676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.035441 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.035450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.036598 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.036619 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.036724 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.036748 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:50:57.036813 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 04:50:57.036880 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":17}
I20260808 04:50:57.037101 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.031337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.15.30.67:46127)
0808 04:50:57.032332 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.032343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.032407 (+ 64us) server_negotiation.cc:407] Connection header received
0808 04:50:57.032485 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.032490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.032696 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.033156 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:57.033305 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.034543 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.035608 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.036474 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.036665 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.036782 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.036841 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.036904 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":114,"threads_started":1}
I20260808 04:50:57.037537 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.031394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:40717 (local address 127.0.0.1:53570)
0808 04:50:57.031708 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:57.031722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:57.031810 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:57.033279 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:57.033281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:57.033296 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:57.033389 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.033394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.034386 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.034391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.035412 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.035422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.036535 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.036538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.037237 (+ 699us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.037266 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:50:57.037338 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:50:57.037409 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260808 04:50:57.038007 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.031895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:40251 (local address 127.0.0.1:56524)
0808 04:50:57.032110 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:57.032124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:57.032215 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:57.033001 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:57.033004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:57.033020 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:57.033168 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.033175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.034796 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.034799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.036293 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.036302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.037405 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.037409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.037508 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.037531 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:50:57.037786 (+ 255us) client_negotiation.cc:240] Negotiation successful
0808 04:50:57.037860 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260808 04:50:57.038089 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.032036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.15.30.66:40251)
0808 04:50:57.032579 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.032589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.032608 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:50:57.032674 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.032689 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.032863 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.033318 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:57.033499 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.034687 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.036434 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.037287 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.037382 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.037513 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.037830 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.037914 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":162,"threads_started":1}
I20260808 04:50:57.038131 27454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.031554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.15.30.65:40717)
0808 04:50:57.032567 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.032627 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.032653 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:50:57.032733 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.032739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.032939 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.033141 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:57.033505 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.034272 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.035546 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.036411 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.036546 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.036738 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.037751 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.037869 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":70,"threads_started":1}
I20260808 04:50:57.040462 27308 tablet_service.cc:1467] Processing CreateTablet for tablet 3112ccff57e44759bdf7c23826cc130a (DEFAULT_TABLE table=test-table [id=9ba7f577bec24c49a341af8a44d47964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:57.040777 26887 tablet_service.cc:1467] Processing CreateTablet for tablet 3112ccff57e44759bdf7c23826cc130a (DEFAULT_TABLE table=test-table [id=9ba7f577bec24c49a341af8a44d47964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:57.041631 26887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3112ccff57e44759bdf7c23826cc130a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:57.041162 27099 tablet_service.cc:1467] Processing CreateTablet for tablet 3112ccff57e44759bdf7c23826cc130a (DEFAULT_TABLE table=test-table [id=9ba7f577bec24c49a341af8a44d47964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:57.041662 27308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3112ccff57e44759bdf7c23826cc130a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:57.042163 27099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3112ccff57e44759bdf7c23826cc130a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:57.049293 27457 tablet_bootstrap.cc:492] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Bootstrap starting.
I20260808 04:50:57.050629 27460 tablet_bootstrap.cc:492] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Bootstrap starting.
I20260808 04:50:57.052205 27457 tablet_bootstrap.cc:654] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:57.052695 27462 tablet_bootstrap.cc:492] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Bootstrap starting.
I20260808 04:50:57.053426 27460 tablet_bootstrap.cc:654] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:57.053478 27457 log.cc:824] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:57.054368 27460 log.cc:824] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:57.055084 27462 tablet_bootstrap.cc:654] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:57.055652 27457 tablet_bootstrap.cc:492] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: No bootstrap required, opened a new log
I20260808 04:50:57.055837 27457 ts_tablet_manager.cc:1397] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:50:57.056141 27462 log.cc:824] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:50:57.056542 27460 tablet_bootstrap.cc:492] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: No bootstrap required, opened a new log
I20260808 04:50:57.056733 27460 ts_tablet_manager.cc:1397] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:50:57.057904 27462 tablet_bootstrap.cc:492] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: No bootstrap required, opened a new log
I20260808 04:50:57.058096 27462 ts_tablet_manager.cc:1397] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 04:50:57.061942 27457 raft_consensus.cc:348] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.062296 27457 raft_consensus.cc:374] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:57.062357 27457 raft_consensus.cc:729] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eaf5c3842aa4ad28cfae11225fd345c, State: Initialized, Role: FOLLOWER
I20260808 04:50:57.062724 27460 raft_consensus.cc:348] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.062955 27460 raft_consensus.cc:374] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:57.063019 27460 raft_consensus.cc:729] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b1338766d540a584b8fd50b4e950b9, State: Initialized, Role: FOLLOWER
I20260808 04:50:57.062981 27457 consensus_queue.cc:260] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.063441 27460 consensus_queue.cc:260] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.064004 27457 ts_tablet_manager.cc:1428] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260808 04:50:57.064385 27460 ts_tablet_manager.cc:1428] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:50:57.065287 27000 heartbeater.cc:503] Master 127.15.30.126:34745 was elected leader, sending a full tablet report...
I20260808 04:50:57.065994 27462 raft_consensus.cc:348] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.066296 27462 raft_consensus.cc:374] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:57.066399 27462 raft_consensus.cc:729] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1a534a60b44186a8a079e9d4c4c0cc, State: Initialized, Role: FOLLOWER
I20260808 04:50:57.066840 27462 consensus_queue.cc:260] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.067812 27462 ts_tablet_manager.cc:1428] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260808 04:50:57.067891 27201 heartbeater.cc:503] Master 127.15.30.126:34745 was elected leader, sending a full tablet report...
I20260808 04:50:57.070839 27408 heartbeater.cc:503] Master 127.15.30.126:34745 was elected leader, sending a full tablet report...
W20260808 04:50:57.233335 27411 tablet.cc:2369] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:50:57.272655 27203 tablet.cc:2369] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:50:57.291505 27002 tablet.cc:2369] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:50:57.347285 27468 raft_consensus.cc:482] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:57.347484 27468 raft_consensus.cc:504] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.348713 27468 leader_election.cc:288] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251), 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717)
I20260808 04:50:57.352389 27454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.349253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:50511 (local address 127.15.30.65:40717)
0808 04:50:57.349383 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.349390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.349405 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:57.349458 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.349460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.349505 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.349583 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:57.349855 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.350353 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.351425 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.351818 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.351941 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.351992 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.352171 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.352243 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:50:57.352449 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.349081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:40717 (local address 127.15.30.67:50511)
0808 04:50:57.349233 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:57.349247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:57.349336 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:57.349603 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:57.349606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:57.349623 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:57.349739 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.349744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.350459 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.350464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.351295 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.351304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.351936 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.351939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.352043 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.352062 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:50:57.352154 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:50:57.352253 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":55}
I20260808 04:50:57.352607 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.349091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:46883 (local address 127.15.30.66:40251)
0808 04:50:57.349308 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.349315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.349332 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:50:57.349392 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.349395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.349434 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.349520 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:57.349821 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.350389 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.351524 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.351956 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.352089 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.352141 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.352349 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.352450 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 04:50:57.352698 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.348918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:40251 (local address 127.15.30.67:46883)
0808 04:50:57.349115 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:57.349133 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:57.349242 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:57.349544 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:57.349547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:57.349565 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:57.349696 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.349701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.350506 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.350513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.351383 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:57.351392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.352089 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.352093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:57.352220 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.352238 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:50:57.352330 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:50:57.352433 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":32}
I20260808 04:50:57.353307 27131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3112ccff57e44759bdf7c23826cc130a" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b1338766d540a584b8fd50b4e950b9" is_pre_election: true
I20260808 04:50:57.353309 26924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3112ccff57e44759bdf7c23826cc130a" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" is_pre_election: true
I20260808 04:50:57.353595 27131 raft_consensus.cc:2454] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 0.
I20260808 04:50:57.353623 26924 raft_consensus.cc:2454] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 0.
I20260808 04:50:57.354055 27231 leader_election.cc:302] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b1338766d540a584b8fd50b4e950b9, cd1a534a60b44186a8a079e9d4c4c0cc; no voters:
I20260808 04:50:57.354326 27468 raft_consensus.cc:2781] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:57.354419 27468 raft_consensus.cc:482] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:57.354452 27468 raft_consensus.cc:3037] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:57.355597 27468 raft_consensus.cc:504] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.355947 27468 leader_election.cc:288] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 election: Requested vote from peers 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251), 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717)
I20260808 04:50:57.356293 26924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3112ccff57e44759bdf7c23826cc130a" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eaf5c3842aa4ad28cfae11225fd345c"
I20260808 04:50:57.356354 27131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3112ccff57e44759bdf7c23826cc130a" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b1338766d540a584b8fd50b4e950b9"
I20260808 04:50:57.356455 26924 raft_consensus.cc:3037] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:57.356509 27131 raft_consensus.cc:3037] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:57.357571 27131 raft_consensus.cc:2454] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 1.
I20260808 04:50:57.357750 26924 raft_consensus.cc:2454] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 1.
I20260808 04:50:57.357904 27231 leader_election.cc:302] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b1338766d540a584b8fd50b4e950b9, cd1a534a60b44186a8a079e9d4c4c0cc; no voters:
I20260808 04:50:57.358161 27468 raft_consensus.cc:2781] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:57.358383 27468 raft_consensus.cc:686] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 LEADER]: Becoming Leader. State: Replica: cd1a534a60b44186a8a079e9d4c4c0cc, State: Running, Role: LEADER
I20260808 04:50:57.358708 27468 consensus_queue.cc:237] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } }
I20260808 04:50:57.362124 25463 catalog_manager.cc:5495] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc reported cstate change: term changed from 0 to 1, leader changed from <none> to cd1a534a60b44186a8a079e9d4c4c0cc (127.15.30.67). New cstate: current_term: 1 leader_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } health_report { overall_health: UNKNOWN } } }
I20260808 04:50:57.400238 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.396420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.15.30.67:46127)
0808 04:50:57.396555 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.396559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.396572 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:50:57.396711 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.396714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.396770 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.396859 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:57.397114 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.397619 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.398648 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.398865 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.398945 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.399309 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:57.399747 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:57.399881 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.399970 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.400027 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:50:57.406559 27131 raft_consensus.cc:1261] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Refusing update from remote peer cd1a534a60b44186a8a079e9d4c4c0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:57.406590 26924 raft_consensus.cc:1261] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Refusing update from remote peer cd1a534a60b44186a8a079e9d4c4c0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:57.407366 27469 consensus_queue.cc:1035] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:57.407569 27468 consensus_queue.cc:1035] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:57.418412 27477 mvcc.cc:205] Tried to move back new op lower bound from 7316130436729606144 to 7316130436543569920. Current Snapshot: MvccSnapshot[applied={T|T < 7316130436729606144}]
I20260808 04:50:57.428599 25464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:57.428750 25464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:57.433928 25464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5a1c13a8d4244699977da3aceb3a1da: Sending DeleteTablet for 3 replicas of tablet 3112ccff57e44759bdf7c23826cc130a
I20260808 04:50:57.434782 26887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3112ccff57e44759bdf7c23826cc130a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:57 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:53570
I20260808 04:50:57.434760 27099 tablet_service.cc:1514] Processing DeleteTablet for tablet 3112ccff57e44759bdf7c23826cc130a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:57 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:56524
I20260808 04:50:57.434768 27308 tablet_service.cc:1514] Processing DeleteTablet for tablet 3112ccff57e44759bdf7c23826cc130a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:57 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:59314
I20260808 04:50:57.435734 27492 tablet_replica.cc:330] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: stopping tablet replica
I20260808 04:50:57.436086 27492 raft_consensus.cc:2229] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:57.437055 27492 raft_consensus.cc:2258] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:57.437479 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 25419
I20260808 04:50:57.454758 27492 ts_tablet_manager.cc:1905] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:57.456574 27491 tablet_replica.cc:330] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: stopping tablet replica
I20260808 04:50:57.457063 27491 raft_consensus.cc:2229] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:57.457399 27494 tablet_replica.cc:330] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: stopping tablet replica
I20260808 04:50:57.457891 27492 ts_tablet_manager.cc:1918] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:57.458001 27492 log.cc:1196] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/3112ccff57e44759bdf7c23826cc130a
I20260808 04:50:57.458366 27492 ts_tablet_manager.cc:1939] T 3112ccff57e44759bdf7c23826cc130a P 2eaf5c3842aa4ad28cfae11225fd345c: Deleting consensus metadata
I20260808 04:50:57.458395 27494 raft_consensus.cc:2229] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:57.458636 27491 raft_consensus.cc:2258] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:57.459023 27494 raft_consensus.cc:2258] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:57.461444 27494 ts_tablet_manager.cc:1905] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 04:50:57.462438 26813 connection.cc:535] server connection from 127.0.0.1:53570 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:50:57.464138 27494 ts_tablet_manager.cc:1918] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:57.464239 27494 log.cc:1196] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/3112ccff57e44759bdf7c23826cc130a
I20260808 04:50:57.464618 27494 ts_tablet_manager.cc:1939] T 3112ccff57e44759bdf7c23826cc130a P 88b1338766d540a584b8fd50b4e950b9: Deleting consensus metadata
W20260808 04:50:57.465652 27032 connection.cc:441] server connection from 127.0.0.1:56524 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56524 (request call id 1) could send its response
I20260808 04:50:57.467065 27491 ts_tablet_manager.cc:1905] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:57.470047 27491 ts_tablet_manager.cc:1918] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:57.470147 27491 log.cc:1196] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/3112ccff57e44759bdf7c23826cc130a
I20260808 04:50:57.470474 27491 ts_tablet_manager.cc:1939] T 3112ccff57e44759bdf7c23826cc130a P cd1a534a60b44186a8a079e9d4c4c0cc: Deleting consensus metadata
W20260808 04:50:57.471745 27235 connection.cc:441] server connection from 127.0.0.1:59314 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59314 (request call id 1) could send its response
I20260808 04:50:57.475548 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.436640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.15.30.125:34027)
0808 04:50:57.436783 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.436788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.460022 (+ 23234us) server_negotiation.cc:407] Connection header received
0808 04:50:57.460403 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.460411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.460484 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.460593 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:57.462000 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.462966 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.465059 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.465439 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.465540 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.467938 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:57.467944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:57.467947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:57.467979 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:57.470068 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.470677 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.470681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.470684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.470792 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.471200 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.471205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.471207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.471483 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.471643 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.475246 (+ 3603us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.475373 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:50:57.481352 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.469065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.15.30.125:34027)
0808 04:50:57.469214 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.469221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.470672 (+ 1451us) server_negotiation.cc:407] Connection header received
0808 04:50:57.470865 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.470872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.470922 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.470978 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:57.472028 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.473032 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.474405 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.474772 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.474848 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.476984 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:57.476988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:57.476991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:57.477016 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:57.479472 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.479962 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.479965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.479967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.480053 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.480356 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.480360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.480363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.480633 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.480743 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.481096 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.481204 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:50:57.482700 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.468767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.15.30.124:40143)
0808 04:50:57.468914 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.468919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.471682 (+ 2763us) server_negotiation.cc:407] Connection header received
0808 04:50:57.471882 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.471885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.471933 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.471988 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:57.473301 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.474089 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.475618 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.475996 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.476077 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.478017 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:57.478024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:57.478031 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:57.478072 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:57.480118 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.480877 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.480884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.480886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.480988 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.481381 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.481387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.481393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.481958 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.482089 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.482411 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.482526 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260808 04:50:57.484027 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:57.437188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.15.30.124:40143)
0808 04:50:57.437307 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:57.437312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:57.472697 (+ 35385us) server_negotiation.cc:407] Connection header received
0808 04:50:57.472748 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:57.472751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:57.472797 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:57.472868 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:50:57.474005 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.474832 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.477321 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:57.477688 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:57.477786 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:57.479589 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:50:57.479594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:50:57.479596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:50:57.479625 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:50:57.481776 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.482305 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.482309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.482310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.482400 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:50:57.482752 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:50:57.482755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:50:57.482759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:50:57.482994 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:50:57.483144 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:57.483766 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 04:50:57.483880 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:50:58.420885 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.420276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.65:50165)
0808 04:50:58.420624 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.420760 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
W20260808 04:50:58.421322 27000 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:34745 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
I20260808 04:50:58.422116 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.421353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:44041)
0808 04:50:58.421916 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.422017 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":67,"threads_started":1}
I20260808 04:50:58.422149 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.421135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.66:45827)
0808 04:50:58.421951 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.422041 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":92,"threads_started":1}
W20260808 04:50:58.422731 27408 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:34745 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
W20260808 04:50:58.422880 27201 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:34745 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
I20260808 04:50:58.933706 27509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5a1c13a8d4244699977da3aceb3a1da)
I20260808 04:50:58.933874 27509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } }
I20260808 04:50:58.934330 27513 raft_consensus.cc:482] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5a1c13a8d4244699977da3aceb3a1da)
I20260808 04:50:58.934422 27509 leader_election.cc:288] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745), fabbd9170046444782edc6cadd9eacda (127.15.30.124:40143)
I20260808 04:50:58.934986 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.934531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37496)
0808 04:50:58.934787 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.934878 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260808 04:50:58.934942 27513 raft_consensus.cc:504] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } }
W20260808 04:50:58.935360 26394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
I20260808 04:50:58.936267 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.936019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37498)
0808 04:50:58.936155 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.936203 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 04:50:58.936431 26394 leader_election.cc:334] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745): Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
I20260808 04:50:58.936620 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.936210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37510)
0808 04:50:58.936441 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.936511 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
W20260808 04:50:58.936973 26697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
I20260808 04:50:58.937147 27513 leader_election.cc:288] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745), 03ce5b0398a04b92b2311e6ab9f2aa5a (127.15.30.125:34027)
I20260808 04:50:58.937983 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.937603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37524)
0808 04:50:58.937840 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.937897 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
W20260808 04:50:58.938843 26697 leader_election.cc:334] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745): Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
I20260808 04:50:58.939462 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.934802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.15.30.124:40143)
0808 04:50:58.935017 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:58.935024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:58.935039 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:58.935093 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:58.935096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:58.935143 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:58.935245 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:58.935568 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.936748 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.938239 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.938877 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.938976 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:58.939050 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:58.939319 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 04:50:58.939380 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260808 04:50:58.939495 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.934630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:40143 (local address 127.0.0.1:59512)
0808 04:50:58.934872 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.934888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:58.934993 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:58.935298 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:58.935301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:58.935323 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:58.935428 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:58.935434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.936852 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.936855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:58.938094 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:58.938105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.939091 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.939095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:58.939187 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:58.939215 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:50:58.939302 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:50:58.939372 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":57,"threads_started":1}
I20260808 04:50:58.939924 26738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fabbd9170046444782edc6cadd9eacda" is_pre_election: true
I20260808 04:50:58.940122 26738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ce5b0398a04b92b2311e6ab9f2aa5a in term 1.
I20260808 04:50:58.940466 26395 leader_election.cc:302] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03ce5b0398a04b92b2311e6ab9f2aa5a, fabbd9170046444782edc6cadd9eacda; no voters: f5a1c13a8d4244699977da3aceb3a1da
I20260808 04:50:58.940639 27509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 04:50:58.940708 27509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader f5a1c13a8d4244699977da3aceb3a1da)
I20260808 04:50:58.940764 27509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:50:58.941315 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.936836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.15.30.125:34027)
0808 04:50:58.937398 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:58.937406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:58.937417 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:50:58.937472 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:58.937477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:58.937532 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:58.937619 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:58.937876 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.938683 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.940162 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.940763 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.940861 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:58.940913 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:58.941154 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 04:50:58.941232 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":115,"threads_started":1}
I20260808 04:50:58.941319 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.936702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:34027 (local address 127.0.0.1:55544)
0808 04:50:58.936947 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.936964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:58.937045 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:58.937645 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:58.937649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:58.937669 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:58.937748 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:58.937754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.938789 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.938791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:58.940011 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:58.940024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:58.940907 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:58.940911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:58.941028 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:58.941052 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:50:58.941142 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:50:58.941212 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":80}
I20260808 04:50:58.941829 26436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fabbd9170046444782edc6cadd9eacda" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" is_pre_election: true
I20260808 04:50:58.942051 27509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } }
I20260808 04:50:58.942310 26436 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fabbd9170046444782edc6cadd9eacda in current term 2: Already voted for candidate 03ce5b0398a04b92b2311e6ab9f2aa5a in this term.
I20260808 04:50:58.942420 27509 leader_election.cc:288] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 2 election: Requested vote from peers f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745), fabbd9170046444782edc6cadd9eacda (127.15.30.124:40143)
I20260808 04:50:58.942735 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.942509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37534)
0808 04:50:58.942600 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.942677 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 04:50:58.942790 26704 leader_election.cc:302] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fabbd9170046444782edc6cadd9eacda; no voters: 03ce5b0398a04b92b2311e6ab9f2aa5a, f5a1c13a8d4244699977da3aceb3a1da
I20260808 04:50:58.943008 27513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:50:58.943368 26738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fabbd9170046444782edc6cadd9eacda"
I20260808 04:50:58.943419 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.943209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37536)
0808 04:50:58.943322 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.943373 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:50:58.943564 26394 leader_election.cc:334] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745): Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
I20260808 04:50:58.944123 27513 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:50:58.945073 26738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ce5b0398a04b92b2311e6ab9f2aa5a in term 2.
I20260808 04:50:58.945374 26395 leader_election.cc:302] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03ce5b0398a04b92b2311e6ab9f2aa5a, fabbd9170046444782edc6cadd9eacda; no voters: f5a1c13a8d4244699977da3aceb3a1da
I20260808 04:50:58.945575 27509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:50:58.945801 27509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [term 2 LEADER]: Becoming Leader. State: Replica: 03ce5b0398a04b92b2311e6ab9f2aa5a, State: Running, Role: LEADER
I20260808 04:50:58.946048 27509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } }
I20260808 04:50:58.947256 27526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03ce5b0398a04b92b2311e6ab9f2aa5a. Latest consensus state: current_term: 2 leader_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:58.947358 27526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:58.947652 27529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:50:58.949831 27529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:50:58.950229 27529 catalog_manager.cc:1259] Loaded cluster ID: eb89cc6dc32e401bbf5d3acafbe85a83
I20260808 04:50:58.950296 27529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:50:58.950824 27529 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:50:58.951259 27529 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Loaded TSK: 0
I20260808 04:50:58.951905 27529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:50:58.986879 26422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:50:58.990162 26422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:50:58.993221 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.992986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37550)
0808 04:50:58.993098 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.993155 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 04:50:58.993248 26738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [term 2 FOLLOWER]: Refusing update from remote peer 03ce5b0398a04b92b2311e6ab9f2aa5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 04:50:58.993868 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:58.993639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.0.0.1:37566)
0808 04:50:58.993740 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:58.993791 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 04:50:58.993760 27526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260808 04:50:58.994011 26394 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a -> Peer f5a1c13a8d4244699977da3aceb3a1da (127.15.30.126:34745): Couldn't send request to peer f5a1c13a8d4244699977da3aceb3a1da. Status: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:50:58.995378 27513 sys_catalog.cc:455] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03ce5b0398a04b92b2311e6ab9f2aa5a. Latest consensus state: current_term: 2 leader_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:58.995534 27513 sys_catalog.cc:458] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:50:58.996532 27526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:58.996652 27526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a [sys.catalog]: This master's current role is: LEADER
I20260808 04:50:58.996806 27513 sys_catalog.cc:455] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5a1c13a8d4244699977da3aceb3a1da" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34745 } } peers { permanent_uuid: "03ce5b0398a04b92b2311e6ab9f2aa5a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "fabbd9170046444782edc6cadd9eacda" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 40143 } attrs { promote: false } } }
I20260808 04:50:58.996937 27513 sys_catalog.cc:458] T 00000000000000000000000000000000 P fabbd9170046444782edc6cadd9eacda [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:50:59.007565 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.002733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:40251 (local address 127.0.0.1:56526)
0808 04:50:59.002822 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.002837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:59.002917 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:59.003614 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:59.003617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:59.003649 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:59.003779 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:59.003786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.004803 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.004806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:59.006241 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:59.006251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.007146 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.007149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:59.007279 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:59.007302 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:50:59.007367 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 04:50:59.007432 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":105}
I20260808 04:50:59.007565 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.003444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.15.30.67:46127)
0808 04:50:59.004032 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:59.004036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:59.004048 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:50:59.004097 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:59.004100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:59.004154 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:59.004246 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:59.004498 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.005138 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.006427 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.006981 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.007104 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:59.007173 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:59.007344 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:50:59.007419 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":79,"threads_started":1}
I20260808 04:50:59.007584 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.003160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:46127 (local address 127.0.0.1:59334)
0808 04:50:59.003695 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.003708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:59.003797 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:59.004273 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:59.004276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:59.004294 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:59.004401 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:59.004407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.005253 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.005258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:59.006283 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:59.006292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.007104 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.007108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:59.007212 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:59.007239 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:50:59.007325 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:50:59.007399 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260808 04:50:59.008283 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.002429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:40717 (local address 127.0.0.1:53578)
0808 04:50:59.002538 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.002554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:50:59.002640 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:50:59.003578 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:50:59.003581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:50:59.003597 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:50:59.003768 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:59.003776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.004890 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.004893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:59.006208 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:50:59.006220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.007145 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.007148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:50:59.008001 (+ 853us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:59.008030 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:50:59.008101 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:50:59.008179 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":885}
I20260808 04:50:59.008320 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.002583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.15.30.65:40717)
0808 04:50:59.003308 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:59.003313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:59.003329 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:50:59.003388 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:59.003391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:59.003445 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:59.003879 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:59.004009 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.004790 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.006362 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.006998 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.007228 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:59.007293 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:59.008136 (+ 843us) server_negotiation.cc:299] Negotiation successful
0808 04:50:59.008221 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":95,"threads_started":1}
I20260808 04:50:59.008279 27308 tablet_service.cc:1467] Processing CreateTablet for tablet c091c842da2c4869a6d65c6da0c9b43f (DEFAULT_TABLE table=test-table [id=7422ed1389984770896277e1f96ae57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:59.008682 27308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c091c842da2c4869a6d65c6da0c9b43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:59.008849 26887 tablet_service.cc:1467] Processing CreateTablet for tablet c091c842da2c4869a6d65c6da0c9b43f (DEFAULT_TABLE table=test-table [id=7422ed1389984770896277e1f96ae57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:59.009145 26887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c091c842da2c4869a6d65c6da0c9b43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:59.011518 27542 tablet_bootstrap.cc:492] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: Bootstrap starting.
I20260808 04:50:59.012041 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.003068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.15.30.66:40251)
0808 04:50:59.003347 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:59.003354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:59.003369 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:50:59.003425 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:59.003427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:59.003474 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:59.003894 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:50:59.003994 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.004683 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.006370 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.007005 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.008570 (+ 1565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:59.011748 (+ 3178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:59.011790 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 04:50:59.011880 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
I20260808 04:50:59.012528 27543 tablet_bootstrap.cc:492] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c: Bootstrap starting.
I20260808 04:50:59.012571 27542 tablet_bootstrap.cc:654] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:59.012566 27099 tablet_service.cc:1467] Processing CreateTablet for tablet c091c842da2c4869a6d65c6da0c9b43f (DEFAULT_TABLE table=test-table [id=7422ed1389984770896277e1f96ae57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:50:59.012846 27099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c091c842da2c4869a6d65c6da0c9b43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:50:59.013435 27543 tablet_bootstrap.cc:654] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:59.014077 27542 tablet_bootstrap.cc:492] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: No bootstrap required, opened a new log
I20260808 04:50:59.014194 27542 ts_tablet_manager.cc:1397] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:50:59.014560 27542 raft_consensus.cc:348] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.014643 27543 tablet_bootstrap.cc:492] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c: No bootstrap required, opened a new log
I20260808 04:50:59.014676 27542 raft_consensus.cc:374] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:59.014717 27543 ts_tablet_manager.cc:1397] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:50:59.014720 27542 raft_consensus.cc:729] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1a534a60b44186a8a079e9d4c4c0cc, State: Initialized, Role: FOLLOWER
I20260808 04:50:59.014843 27542 consensus_queue.cc:260] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.015187 27542 ts_tablet_manager.cc:1428] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:59.015225 27543 raft_consensus.cc:348] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.015326 27543 raft_consensus.cc:374] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:59.015367 27543 raft_consensus.cc:729] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eaf5c3842aa4ad28cfae11225fd345c, State: Initialized, Role: FOLLOWER
I20260808 04:50:59.015738 27543 consensus_queue.cc:260] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.016566 27543 ts_tablet_manager.cc:1428] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:50:59.017554 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.017153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.65:32847)
0808 04:50:59.017389 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.017446 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20260808 04:50:59.017637 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.015963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:34427)
0808 04:50:59.017487 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.017542 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1462,"thread_start_us":98,"threads_started":1}
I20260808 04:50:59.018020 27410 heartbeater.cc:503] Master 127.15.30.125:34027 was elected leader, sending a full tablet report...
I20260808 04:50:59.018869 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.018266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:39119)
0808 04:50:59.018717 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.018778 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":94,"threads_started":1}
I20260808 04:50:59.018859 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.018592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.65:45347)
0808 04:50:59.018745 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.018795 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 04:50:59.019037 27001 heartbeater.cc:503] Master 127.15.30.125:34027 was elected leader, sending a full tablet report...
I20260808 04:50:59.020170 27550 tablet_bootstrap.cc:492] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: Bootstrap starting.
I20260808 04:50:59.020892 27550 tablet_bootstrap.cc:654] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: Neither blocks nor log segments found. Creating new log.
I20260808 04:50:59.021982 27550 tablet_bootstrap.cc:492] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: No bootstrap required, opened a new log
I20260808 04:50:59.022086 27550 ts_tablet_manager.cc:1397] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 04:50:59.022583 27550 raft_consensus.cc:348] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.022694 27550 raft_consensus.cc:374] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:50:59.022739 27550 raft_consensus.cc:729] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b1338766d540a584b8fd50b4e950b9, State: Initialized, Role: FOLLOWER
I20260808 04:50:59.022866 27550 consensus_queue.cc:260] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.023159 27550 ts_tablet_manager.cc:1428] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:50:59.023944 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.023508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.66:52675)
0808 04:50:59.023803 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.023858 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260808 04:50:59.024555 27202 heartbeater.cc:503] Master 127.15.30.125:34027 was elected leader, sending a full tablet report...
I20260808 04:50:59.024576 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.024359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.66:43619)
0808 04:50:59.024477 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.024522 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:50:59.116585 27546 raft_consensus.cc:482] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:50:59.116739 27546 raft_consensus.cc:504] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.117249 27546 leader_election.cc:288] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717), 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251)
I20260808 04:50:59.117755 27131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091c842da2c4869a6d65c6da0c9b43f" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b1338766d540a584b8fd50b4e950b9" is_pre_election: true
I20260808 04:50:59.117755 26924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091c842da2c4869a6d65c6da0c9b43f" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" is_pre_election: true
I20260808 04:50:59.117935 27131 raft_consensus.cc:2454] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 0.
I20260808 04:50:59.117934 26924 raft_consensus.cc:2454] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 0.
I20260808 04:50:59.118291 27231 leader_election.cc:302] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2eaf5c3842aa4ad28cfae11225fd345c, 88b1338766d540a584b8fd50b4e950b9, cd1a534a60b44186a8a079e9d4c4c0cc; no voters:
I20260808 04:50:59.118485 27546 raft_consensus.cc:2781] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:50:59.118553 27546 raft_consensus.cc:482] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:50:59.118582 27546 raft_consensus.cc:3037] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:59.119650 27546 raft_consensus.cc:504] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.119997 27546 leader_election.cc:288] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 election: Requested vote from peers 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717), 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251)
I20260808 04:50:59.120357 26924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091c842da2c4869a6d65c6da0c9b43f" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eaf5c3842aa4ad28cfae11225fd345c"
I20260808 04:50:59.120455 27131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091c842da2c4869a6d65c6da0c9b43f" candidate_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b1338766d540a584b8fd50b4e950b9"
I20260808 04:50:59.120535 26924 raft_consensus.cc:3037] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:59.120602 27131 raft_consensus.cc:3037] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:50:59.121519 27131 raft_consensus.cc:2454] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 1.
I20260808 04:50:59.121533 26924 raft_consensus.cc:2454] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd1a534a60b44186a8a079e9d4c4c0cc in term 1.
I20260808 04:50:59.121836 27231 leader_election.cc:302] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b1338766d540a584b8fd50b4e950b9, cd1a534a60b44186a8a079e9d4c4c0cc; no voters:
I20260808 04:50:59.122117 27546 raft_consensus.cc:2781] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:50:59.122362 27546 raft_consensus.cc:686] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 LEADER]: Becoming Leader. State: Replica: cd1a534a60b44186a8a079e9d4c4c0cc, State: Running, Role: LEADER
I20260808 04:50:59.122524 27546 consensus_queue.cc:237] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } }
I20260808 04:50:59.123090 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.122836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:58405)
0808 04:50:59.122943 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.123005 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 04:50:59.124246 26421 catalog_manager.cc:5495] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc reported cstate change: term changed from 0 to 1, leader changed from <none> to cd1a534a60b44186a8a079e9d4c4c0cc (127.15.30.67). New cstate: current_term: 1 leader_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd1a534a60b44186a8a079e9d4c4c0cc" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46127 } health_report { overall_health: HEALTHY } } }
I20260808 04:50:59.160199 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.157236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.15.30.67:46127)
0808 04:50:59.157362 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:50:59.157366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:50:59.157378 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:50:59.157423 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:50:59.157425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:50:59.157467 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:50:59.157531 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:50:59.157817 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.158323 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.159175 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:50:59.159383 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:50:59.159454 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:50:59.159708 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:50:59.159779 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:50:59.159843 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:50:59.159992 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 04:50:59.160081 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:50:59.162428 26924 raft_consensus.cc:1261] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Refusing update from remote peer cd1a534a60b44186a8a079e9d4c4c0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:59.162428 27131 raft_consensus.cc:1261] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Refusing update from remote peer cd1a534a60b44186a8a079e9d4c4c0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:50:59.162910 27546 consensus_queue.cc:1035] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eaf5c3842aa4ad28cfae11225fd345c" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:59.163148 27564 consensus_queue.cc:1035] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b1338766d540a584b8fd50b4e950b9" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 40251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:50:59.166653 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.166291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.65:35783)
0808 04:50:59.166509 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.166582 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260808 04:50:59.166970 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.164773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.66:44875)
0808 04:50:59.166837 (+ 2064us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.166896 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1990}
I20260808 04:50:59.167505 27569 mvcc.cc:205] Tried to move back new op lower bound from 7316130443924860928 to 7316130443767312384. Current Snapshot: MvccSnapshot[applied={T|T < 7316130443924860928}]
I20260808 04:50:59.168314 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.168000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:35129)
0808 04:50:59.168175 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.168247 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 04:50:59.169836 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.169557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.67:34659)
0808 04:50:59.169713 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.169768 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260808 04:50:59.170542 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.170269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.65:45193)
0808 04:50:59.170419 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.170474 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 04:50:59.171825 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.171513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34745 (local address 127.15.30.66:34201)
0808 04:50:59.171671 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.171726 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:34745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 04:50:59.171748 27570 mvcc.cc:205] Tried to move back new op lower bound from 7316130443924860928 to 7316130443767312384. Current Snapshot: MvccSnapshot[applied={T|T < 7316130443924860928}]
I20260808 04:50:59.175836 26421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:59.176023 26421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:50:59.179701 26421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03ce5b0398a04b92b2311e6ab9f2aa5a: Sending DeleteTablet for 3 replicas of tablet c091c842da2c4869a6d65c6da0c9b43f
I20260808 04:50:59.180307 26887 tablet_service.cc:1514] Processing DeleteTablet for tablet c091c842da2c4869a6d65c6da0c9b43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:59 UTC) from {username='oryx', principal='oryx/127.15.30.125@KRBTEST.COM'} at 127.0.0.1:53578
I20260808 04:50:59.180622 27099 tablet_service.cc:1514] Processing DeleteTablet for tablet c091c842da2c4869a6d65c6da0c9b43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:59 UTC) from {username='oryx', principal='oryx/127.15.30.125@KRBTEST.COM'} at 127.0.0.1:56526
I20260808 04:50:59.180716 27584 tablet_replica.cc:330] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c: stopping tablet replica
I20260808 04:50:59.180846 27584 raft_consensus.cc:2229] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:59.180913 27308 tablet_service.cc:1514] Processing DeleteTablet for tablet c091c842da2c4869a6d65c6da0c9b43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:50:59 UTC) from {username='oryx', principal='oryx/127.15.30.125@KRBTEST.COM'} at 127.0.0.1:59334
I20260808 04:50:59.180985 27584 raft_consensus.cc:2258] T c091c842da2c4869a6d65c6da0c9b43f P 2eaf5c3842aa4ad28cfae11225fd345c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:59.181072 27585 tablet_replica.cc:330] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: stopping tablet replica
I20260808 04:50:59.181210 27586 tablet_replica.cc:330] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: stopping tablet replica
I20260808 04:50:59.181290 27585 raft_consensus.cc:2229] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:50:59.181306 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 26654
I20260808 04:50:59.181342 27586 raft_consensus.cc:2229] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:50:59.184252 27586 raft_consensus.cc:2258] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:59.186419 27586 ts_tablet_manager.cc:1905] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:59.188892 27586 ts_tablet_manager.cc:1918] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:59.188977 27586 log.cc:1196] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/c091c842da2c4869a6d65c6da0c9b43f
I20260808 04:50:59.189368 27586 ts_tablet_manager.cc:1939] T c091c842da2c4869a6d65c6da0c9b43f P cd1a534a60b44186a8a079e9d4c4c0cc: Deleting consensus metadata
I20260808 04:50:59.190290 27585 raft_consensus.cc:2258] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:50:59.192073 27585 ts_tablet_manager.cc:1905] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:50:59.195765 27585 ts_tablet_manager.cc:1918] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:50:59.195860 27585 log.cc:1196] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/c091c842da2c4869a6d65c6da0c9b43f
I20260808 04:50:59.196157 27585 ts_tablet_manager.cc:1939] T c091c842da2c4869a6d65c6da0c9b43f P 88b1338766d540a584b8fd50b4e950b9: Deleting consensus metadata
I20260808 04:50:59.196830 26394 catalog_manager.cc:4841] TS cd1a534a60b44186a8a079e9d4c4c0cc (127.15.30.67:46127): tablet c091c842da2c4869a6d65c6da0c9b43f (table test-table [id=7422ed1389984770896277e1f96ae57b]) successfully deleted
I20260808 04:50:59.196944 26395 catalog_manager.cc:4841] TS 88b1338766d540a584b8fd50b4e950b9 (127.15.30.66:40251): tablet c091c842da2c4869a6d65c6da0c9b43f (table test-table [id=7422ed1389984770896277e1f96ae57b]) successfully deleted
I20260808 04:50:59.201654 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 27008
I20260808 04:50:59.203464 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:50:59.201754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:40717 (local address 127.0.0.1:53590)
0808 04:50:59.203285 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0808 04:50:59.203356 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1428}
W20260808 04:50:59.203751 26401 catalog_manager.cc:4584] TS 2eaf5c3842aa4ad28cfae11225fd345c (127.15.30.65:40717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c091c842da2c4869a6d65c6da0c9b43f: Network error: Client connection negotiation failed: client connection to 127.15.30.65:40717: connect: Connection refused (error 111)
I20260808 04:50:59.211315 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 27208
I20260808 04:50:59.220100 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 26199
I20260808 04:50:59.230271 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 26374
2026-08-08T04:50:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5846 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:50:59 dist-test-slave-zpvx krb5kdc[27592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:50:59 dist-test-slave-zpvx krb5kdc[27592](info): set up 2 sockets
Aug 08 04:50:59 dist-test-slave-zpvx krb5kdc[27592](info): commencing operation
krb5kdc: starting...
W20260808 04:51:00.483378 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:51:00 dist-test-slave-zpvx krb5kdc[27592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:51:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab.
Entry for principal oryx/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab.
I20260808 04:51:00.609205 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33285
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab
--principal=oryx/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:00.722993 27610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:00.723255 27610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:00.723296 27610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:00.727154 27610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:51:00.727202 27610 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:51:00.727221 27610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:00.727238 27610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:00.727254 27610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:51:00.727269 27610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:51:00.732347 27610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33285
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.126.keytab
--principal=oryx/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:00.733333 27610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:00.734293 27610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:00.742583 27618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:00.742651 27620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:00.742666 27617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:00.742878 27610 server_base.cc:1034] running on GCE node
I20260808 04:51:00.743585 27610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:00.744763 27610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:00.745954 27610 hybrid_clock.cc:648] HybridClock initialized: now 1786164660745910 us; error 55 us; skew 500 ppm
Aug 08 04:51:00 dist-test-slave-zpvx krb5kdc[27592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164660, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:00.754782 27610 init.cc:377] Logged in from keytab as oryx/127.15.30.126@KRBTEST.COM (short username oryx)
I20260808 04:51:00.755793 27610 webserver.cc:460] Webserver started at http://127.15.30.126:35943/ using document root <none> and password file <none>
I20260808 04:51:00.756327 27610 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:00.756445 27610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:00.756655 27610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:00.758674 27610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "e88ffa9b72dc4653b88477fe180a6c73"
format_stamp: "Formatted at 2026-08-08 04:51:00 on dist-test-slave-zpvx"
server_key: "5dabdb8d313419ce19b17954790dec1a"
server_key_iv: "2d89d1b28c3061f3df1fc976129640bf"
server_key_version: "encryptionkey@0"
I20260808 04:51:00.759259 27610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "e88ffa9b72dc4653b88477fe180a6c73"
format_stamp: "Formatted at 2026-08-08 04:51:00 on dist-test-slave-zpvx"
server_key: "5dabdb8d313419ce19b17954790dec1a"
server_key_iv: "2d89d1b28c3061f3df1fc976129640bf"
server_key_version: "encryptionkey@0"
I20260808 04:51:00.763052 27610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260808 04:51:00.765543 27630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:00.766804 27610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:51:00.766907 27610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "e88ffa9b72dc4653b88477fe180a6c73"
format_stamp: "Formatted at 2026-08-08 04:51:00 on dist-test-slave-zpvx"
server_key: "5dabdb8d313419ce19b17954790dec1a"
server_key_iv: "2d89d1b28c3061f3df1fc976129640bf"
server_key_version: "encryptionkey@0"
I20260808 04:51:00.766991 27610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:00.803355 27610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:00.806314 27610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:00.806473 27610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:00.814649 27610 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33285
I20260808 04:51:00.814677 27719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33285 every 8 connection(s)
I20260808 04:51:00.815646 27610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:51:00.818555 27721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:00.824216 27721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Bootstrap starting.
I20260808 04:51:00.825271 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 27610
I20260808 04:51:00.825372 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:51:00.825663 15481 external_mini_cluster.cc:1428] Setting key 7781f1a71b1e33e4339b537e5327c630
I20260808 04:51:00.827006 27721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:00.827930 27721 log.cc:824] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:00.830139 27721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: No bootstrap required, opened a new log
I20260808 04:51:00.834287 27721 raft_consensus.cc:348] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88ffa9b72dc4653b88477fe180a6c73" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33285 } }
I20260808 04:51:00.834513 27721 raft_consensus.cc:374] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:00.834591 27721 raft_consensus.cc:729] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88ffa9b72dc4653b88477fe180a6c73, State: Initialized, Role: FOLLOWER
Aug 08 04:51:00 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:51:00.835078 27721 consensus_queue.cc:260] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88ffa9b72dc4653b88477fe180a6c73" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33285 } }
I20260808 04:51:00.835215 27721 raft_consensus.cc:388] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:51:00.835299 27721 raft_consensus.cc:482] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:51:00.835413 27721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:00.836544 27721 raft_consensus.cc:504] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88ffa9b72dc4653b88477fe180a6c73" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33285 } }
I20260808 04:51:00.836886 27721 leader_election.cc:302] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e88ffa9b72dc4653b88477fe180a6c73; no voters:
I20260808 04:51:00.837242 27721 leader_election.cc:288] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:51:00.837306 27729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:00.837520 27729 raft_consensus.cc:686] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [term 1 LEADER]: Becoming Leader. State: Replica: e88ffa9b72dc4653b88477fe180a6c73, State: Running, Role: LEADER
I20260808 04:51:00.837822 27729 consensus_queue.cc:237] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88ffa9b72dc4653b88477fe180a6c73" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33285 } }
I20260808 04:51:00.838506 27721 sys_catalog.cc:564] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:51:00.839570 27731 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e88ffa9b72dc4653b88477fe180a6c73. Latest consensus state: current_term: 1 leader_uuid: "e88ffa9b72dc4653b88477fe180a6c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88ffa9b72dc4653b88477fe180a6c73" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33285 } } }
I20260808 04:51:00.839711 27731 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:00.840041 27732 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e88ffa9b72dc4653b88477fe180a6c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88ffa9b72dc4653b88477fe180a6c73" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33285 } } }
I20260808 04:51:00.840147 27732 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73 [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:00.840502 27743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:51:00.841434 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:00.826759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.15.30.126:33285)
0808 04:51:00.827281 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:00.827304 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:00.827351 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:51:00.828175 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:00.828200 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:00.828628 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:00.828979 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:00.829687 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:00.831134 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:00.832025 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:00.832780 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:00.832942 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:00.835553 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:00.835560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:00.835579 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:00.835607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:00.837604 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:00.838124 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:00.838129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:00.838135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:00.838270 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:00.838553 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:00.838556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:00.838558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:00.839103 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:00.840006 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:00.840170 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 04:51:00.840499 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":348,"thread_start_us":158,"threads_started":1}
I20260808 04:51:00.843722 27743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:51:00.849350 27743 catalog_manager.cc:1347] Generated new cluster ID: 619b221d197c47f6894bbcb3b279d743
I20260808 04:51:00.849422 27743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:51:00.861441 27743 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:51:00.862640 27743 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:51:00.893714 27743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Generated new TSK 0
I20260808 04:51:00.894769 27743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab.
Entry for principal oryx/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab.
I20260808 04:51:00.943871 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33285
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab
--principal=oryx/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:01.066903 27768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:01.067178 27768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:01.067225 27768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:01.071494 27768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:01.071557 27768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:01.071599 27768 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:51:01.071682 27768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:51:01.077059 27768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab
--principal=oryx/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:33285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:01.078188 27768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:01.079275 27768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:01.089321 27778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:01.089375 27781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:01.089324 27783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:01.089700 27768 server_base.cc:1034] running on GCE node
I20260808 04:51:01.090229 27768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:01.090910 27768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:01.092095 27768 hybrid_clock.cc:648] HybridClock initialized: now 1786164661092073 us; error 34 us; skew 500 ppm
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:01.095106 27768 init.cc:377] Logged in from keytab as oryx/127.15.30.65@KRBTEST.COM (short username oryx)
I20260808 04:51:01.096040 27768 webserver.cc:460] Webserver started at http://127.15.30.65:35607/ using document root <none> and password file <none>
I20260808 04:51:01.096683 27768 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:01.096778 27768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:01.097013 27768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:01.099093 27768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "c8c213451a804d5aa332b236ff079df7"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "643ebd5d46e29f4d94ea3539bf54d751"
server_key_iv: "b60d912bc9fb51c88783d0cff8ccb5c3"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.099690 27768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "c8c213451a804d5aa332b236ff079df7"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "643ebd5d46e29f4d94ea3539bf54d751"
server_key_iv: "b60d912bc9fb51c88783d0cff8ccb5c3"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.103488 27768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:51:01.105947 27791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:01.107149 27768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:51:01.107288 27768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "c8c213451a804d5aa332b236ff079df7"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "643ebd5d46e29f4d94ea3539bf54d751"
server_key_iv: "b60d912bc9fb51c88783d0cff8ccb5c3"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.107395 27768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:01.119982 27768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:01.122893 27768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:01.123090 27768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:01.124747 27768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:01.124811 27768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:01.124869 27768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:01.124892 27768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:51:01.136109 27768 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:46829
I20260808 04:51:01.136168 27959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:46829 every 8 connection(s)
I20260808 04:51:01.137311 27768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:51:01.138603 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.125585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:36645 (local address 127.15.30.126:33285)
0808 04:51:01.125721 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.125725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.126663 (+ 938us) server_negotiation.cc:407] Connection header received
0808 04:51:01.127586 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.127589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.127649 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.127738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.128953 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.129708 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.130924 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.131173 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.131277 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.134585 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:01.134589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:01.134594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:01.134630 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:01.136315 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.136889 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.136896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.136901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.136998 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.137317 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.137322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.137327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.137555 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:01.137672 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.138263 (+ 591us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.138472 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:51:01.139416 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.126045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33285 (local address 127.15.30.65:36645)
0808 04:51:01.126521 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.126561 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.127382 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.127915 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.127925 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.128290 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:01.128806 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.128821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.129841 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.129845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.130800 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.130810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.131329 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.131333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.131545 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.132161 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:01.132183 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:01.134408 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:01.136468 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.136474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.136491 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.136780 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.137089 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.137091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.137093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.137218 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.137696 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:01.137703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:01.137970 (+ 267us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.138178 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.138494 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":146,"threads_started":1}
I20260808 04:51:01.140605 27960 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33285
I20260808 04:51:01.140825 27960 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:01.141374 27960 heartbeater.cc:511] Master 127.15.30.126:33285 requested a full tablet report, sending...
I20260808 04:51:01.141868 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 27768
I20260808 04:51:01.142010 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:51:01.142325 15481 external_mini_cluster.cc:1428] Setting key 4e1497776cc8b567bec01f13957efd7b
I20260808 04:51:01.143078 27650 ts_manager.cc:194] Registered new tserver with Master: c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829)
I20260808 04:51:01.144451 27650 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:36645
I20260808 04:51:01.151831 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.143032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:51359 (local address 127.15.30.126:33285)
0808 04:51:01.143589 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.143593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.143607 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:51:01.143639 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.143642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.143677 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.143771 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.144315 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.145046 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.146173 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.146499 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.146583 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.148064 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:01.148067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:01.148069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:01.148094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:01.149919 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.150472 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.150475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.150476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.150556 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.150919 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.150922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.150924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.151113 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:01.151210 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.151593 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.151703 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":484}
I20260808 04:51:01.151981 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.142905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33285 (local address 127.15.30.65:51359)
0808 04:51:01.143158 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.143170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.143241 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.143808 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.143812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.144012 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:01.144133 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.144140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.145168 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.145172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.146013 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.146026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.146743 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.146746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.146837 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.147413 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:01.147425 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:01.147905 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:01.150045 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.150050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.150054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.150364 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.150657 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.150664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.150668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.150782 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.151228 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:01.151233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:01.151375 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.151511 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.151702 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":9,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab.
Entry for principal oryx/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab.
I20260808 04:51:01.173697 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33285
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab
--principal=oryx/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:01.285640 27973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:01.285916 27973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:01.285990 27973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:01.290015 27973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:01.290110 27973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:01.290177 27973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:51:01.290256 27973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:51:01.295435 27973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab
--principal=oryx/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:33285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:01.296502 27973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:01.297533 27973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:01.307123 27983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:01.307132 27987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:01.307210 27980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:01.307206 27973 server_base.cc:1034] running on GCE node
I20260808 04:51:01.307742 27973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:01.308419 27973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:01.309607 27973 hybrid_clock.cc:648] HybridClock initialized: now 1786164661309573 us; error 47 us; skew 500 ppm
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:01.312515 27973 init.cc:377] Logged in from keytab as oryx/127.15.30.66@KRBTEST.COM (short username oryx)
I20260808 04:51:01.313306 27973 webserver.cc:460] Webserver started at http://127.15.30.66:46789/ using document root <none> and password file <none>
I20260808 04:51:01.313817 27973 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:01.313875 27973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:01.314041 27973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:01.315999 27973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "a3f9a8a059d84f6a9b8b40c559632c12"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "ee9f2970f3062ee6d1809b264de57b69"
server_key_iv: "3f868329260f8276f0229c6b5731adff"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.316570 27973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "a3f9a8a059d84f6a9b8b40c559632c12"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "ee9f2970f3062ee6d1809b264de57b69"
server_key_iv: "3f868329260f8276f0229c6b5731adff"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.320106 27973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:51:01.322489 27996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:01.323683 27973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:51:01.323812 27973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "a3f9a8a059d84f6a9b8b40c559632c12"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "ee9f2970f3062ee6d1809b264de57b69"
server_key_iv: "3f868329260f8276f0229c6b5731adff"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.323915 27973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:01.350757 27973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:01.353777 27973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:01.353981 27973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:01.355439 27973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:01.355510 27973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:01.355580 27973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:01.355626 27973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:51:01.366777 27973 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:42389
I20260808 04:51:01.366888 28173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:42389 every 8 connection(s)
I20260808 04:51:01.367802 27973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:51:01.369596 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.356309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:50299 (local address 127.15.30.126:33285)
0808 04:51:01.356505 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.356513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.357382 (+ 869us) server_negotiation.cc:407] Connection header received
0808 04:51:01.358380 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.358383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.358443 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.358504 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.360070 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.360693 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.361838 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.362063 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.362168 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.365692 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:01.365697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:01.365700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:01.365728 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:01.367288 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.367893 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.367898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.367903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.367987 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.368430 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.368435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.368438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.368696 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:01.368812 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.369306 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.369426 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:51:01.369737 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 27973
I20260808 04:51:01.369858 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:51:01.370201 15481 external_mini_cluster.cc:1428] Setting key c4b5035ad92c04ccfbaab10c67cf5143
I20260808 04:51:01.370461 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.356750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33285 (local address 127.15.30.66:50299)
0808 04:51:01.357216 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.357266 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.358131 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.358934 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.358941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.359268 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:01.359858 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.359874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.360828 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.360831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.361713 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.361722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.362222 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.362226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.362406 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.363039 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:01.363061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:01.365492 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:01.367396 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.367404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.367421 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.367767 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.368078 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.368082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.368087 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.368282 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.368809 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:01.368816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:01.369065 (+ 249us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.369257 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.369584 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":127,"threads_started":1}
I20260808 04:51:01.371701 28174 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33285
I20260808 04:51:01.371874 28174 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:01.372506 28174 heartbeater.cc:511] Master 127.15.30.126:33285 requested a full tablet report, sending...
I20260808 04:51:01.373689 27649 ts_manager.cc:194] Registered new tserver with Master: a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389)
I20260808 04:51:01.374434 27649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:50299
WARNING: no policy specified for oryx/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:51:01.383108 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.375083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:48509 (local address 127.15.30.126:33285)
0808 04:51:01.375201 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.375207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.375376 (+ 169us) server_negotiation.cc:407] Connection header received
0808 04:51:01.375528 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.375531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.375584 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.375662 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.376215 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.376856 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.378170 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.378378 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.378453 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.379843 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:01.379846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:01.379848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:01.379870 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:01.381450 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.381938 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.381941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.381942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.382014 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.382305 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.382307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.382308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.382531 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:01.382625 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.382908 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.382999 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:51:01.383116 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.374929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33285 (local address 127.15.30.66:48509)
0808 04:51:01.375296 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.375308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.375426 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.375679 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.375682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.375922 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:01.376083 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.376095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.376985 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.376992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.378038 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.378053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.378506 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.378513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.378643 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.379195 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:01.379214 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:01.379706 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:01.381562 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.381566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.381568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.381837 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.382109 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.382112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.382114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.382214 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.382620 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:01.382624 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:01.382732 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.382848 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.382965 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab.
Entry for principal oryx/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab.
I20260808 04:51:01.401433 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33285
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab
--principal=oryx/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:01.514801 28184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:01.515064 28184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:01.515147 28184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:01.519126 28184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:01.519187 28184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:01.519227 28184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:51:01.519260 28184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:51:01.524710 28184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab
--principal=oryx/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.30.126:33285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:01.525767 28184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:01.526777 28184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:01.536628 28192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:01.536840 28184 server_base.cc:1034] running on GCE node
W20260808 04:51:01.536650 28190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:01.536985 28195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:01.537358 28184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:01.537992 28184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:01.539217 28184 hybrid_clock.cc:648] HybridClock initialized: now 1786164661539187 us; error 40 us; skew 500 ppm
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:01.542250 28184 init.cc:377] Logged in from keytab as oryx/127.15.30.67@KRBTEST.COM (short username oryx)
I20260808 04:51:01.543151 28184 webserver.cc:460] Webserver started at http://127.15.30.67:36369/ using document root <none> and password file <none>
I20260808 04:51:01.543676 28184 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:01.543754 28184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:01.543921 28184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:01.545959 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "16770a1d4516461bb4a6ec62ace82a18"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "af8c328f30c2503a17423722f3628fe1"
server_key_iv: "84c02be8a914a0ee8b793af53bd9f9c0"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.546466 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "16770a1d4516461bb4a6ec62ace82a18"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "af8c328f30c2503a17423722f3628fe1"
server_key_iv: "84c02be8a914a0ee8b793af53bd9f9c0"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.550168 28184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:51:01.552556 28203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:01.553774 28184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:01.553905 28184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "16770a1d4516461bb4a6ec62ace82a18"
format_stamp: "Formatted at 2026-08-08 04:51:01 on dist-test-slave-zpvx"
server_key: "af8c328f30c2503a17423722f3628fe1"
server_key_iv: "84c02be8a914a0ee8b793af53bd9f9c0"
server_key_version: "encryptionkey@0"
I20260808 04:51:01.554006 28184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:01.587285 28184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:01.590369 28184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:01.590583 28184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:01.592087 28184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:01.592162 28184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:01.592226 28184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:01.592273 28184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:51:01.603423 28184 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:45685
I20260808 04:51:01.603533 28382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:45685 every 8 connection(s)
I20260808 04:51:01.604334 28184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:51:01.606746 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.593071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:42189 (local address 127.15.30.126:33285)
0808 04:51:01.593204 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.593208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.594327 (+ 1119us) server_negotiation.cc:407] Connection header received
0808 04:51:01.595390 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.595396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.595466 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.595530 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.597349 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.598250 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.599648 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.599876 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.599989 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.602974 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:01.602978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:01.602980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:01.603014 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:01.604672 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.605256 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.605260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.605262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.605348 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.605681 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.605685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.605687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.605870 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:01.605975 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.606444 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.606563 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:51:01.607478 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.593602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33285 (local address 127.15.30.67:42189)
0808 04:51:01.594142 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.594187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.595111 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.596025 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.596036 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.596531 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:01.597190 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.597213 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.598394 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.598397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.599511 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.599528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.600042 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.600046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.600220 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.601394 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:01.601410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:01.602810 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:01.604785 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.604791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.604807 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.605145 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.605445 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.605448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.605450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.605586 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.605974 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:01.605982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:01.606215 (+ 233us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.606397 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.606707 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":132,"threads_started":1}
I20260808 04:51:01.608306 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 28184
I20260808 04:51:01.608470 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:51:01.608819 15481 external_mini_cluster.cc:1428] Setting key 85a618a51ae87a103d681d08d948a5cb
I20260808 04:51:01.608798 28384 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33285
I20260808 04:51:01.608963 28384 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:01.609562 28384 heartbeater.cc:511] Master 127.15.30.126:33285 requested a full tablet report, sending...
I20260808 04:51:01.610702 27649 ts_manager.cc:194] Registered new tserver with Master: 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685)
I20260808 04:51:01.611475 27649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:42189
I20260808 04:51:01.618016 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.611214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33285 (local address 127.15.30.67:52991)
0808 04:51:01.611465 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.611481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.611571 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.612153 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.612156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.612442 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:01.612550 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.612557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.613317 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.613320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.613923 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.613932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.614371 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.614373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.614443 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.614822 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:01.614832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:01.615120 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:01.616770 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.616773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.616774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.616961 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.617240 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:01.617242 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:01.617243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:01.617315 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:01.617641 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:01.617643 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:01.617712 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.617787 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.617882 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20260808 04:51:01.618023 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.611800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:52991 (local address 127.15.30.126:33285)
0808 04:51:01.611892 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.611895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.611906 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:51:01.611932 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.611934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.611966 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.612048 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.612662 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.613184 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.614027 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.614272 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.614351 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.615238 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:01.615241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:01.615243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:01.615264 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:01.616674 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.617049 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.617052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.617053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.617160 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.617407 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.617409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.617410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.617558 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:01.617633 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.617840 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.617923 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 04:51:01.622609 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:51:01.650491 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:01.644188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.15.30.126:33285)
0808 04:51:01.644331 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.644335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.644350 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:51:01.644556 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.644559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.644604 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.644686 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.645378 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.645939 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.646753 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.646976 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.647061 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.650334 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56336: BlockingRecv error: recv got EOF from 127.0.0.1:56336 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 04:51:01 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164661, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:51:01.659917 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.652078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.15.30.126:33285)
0808 04:51:01.652214 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.652218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.652236 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:51:01.652357 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.652361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.652432 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.652510 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:01.653118 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.653679 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.654454 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.654685 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.654779 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.657107 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:01.657110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:01.657112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:01.657134 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:01.658617 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.658987 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.658989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.658991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.659065 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:01.659310 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:01.659312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:01.659313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:01.659469 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:01.659582 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.659722 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.659804 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:51:01.662842 27649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:51:01.666024 27649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:51:01.680999 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.674603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:45685 (local address 127.0.0.1:39584)
0808 04:51:01.675178 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.675194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.675275 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.676133 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.676139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.676161 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:01.676423 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.676439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.677896 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.677898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.679335 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.679346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.680466 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.680471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.680575 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.680628 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.680738 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.680851 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":83,"thread_start_us":101,"threads_started":1}
I20260808 04:51:01.681056 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.674764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.15.30.67:45685)
0808 04:51:01.675550 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.675564 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.675591 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:51:01.675740 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.675744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.675946 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.676346 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:01.676548 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.677766 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.679475 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.680310 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.680467 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.680638 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.680788 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.680902 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":122,"threads_started":1}
I20260808 04:51:01.681730 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.675923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.15.30.65:46829)
0808 04:51:01.676637 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.676642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.676663 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:51:01.676724 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.676727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.676868 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.677019 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:01.677193 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.678003 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.679550 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.680310 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.680467 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.680630 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.681451 (+ 821us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.681557 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":92,"threads_started":1}
I20260808 04:51:01.681969 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.675516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46829 (local address 127.0.0.1:39694)
0808 04:51:01.675984 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.676000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.676088 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.676997 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.677000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.677017 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:01.677102 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.677109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.678324 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.678327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.679428 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.679438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.680689 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.680693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.680790 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.680815 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.680901 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.680969 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":99,"threads_started":1}
I20260808 04:51:01.682600 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.674360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:42389 (local address 127.0.0.1:38762)
0808 04:51:01.674942 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.674993 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.675132 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.676249 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.676252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.676266 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:01.676413 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.676421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.677891 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.677894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.680204 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.680213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.682234 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.682238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.682324 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.682345 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.682414 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.682481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260808 04:51:01.682660 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.675038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.15.30.66:42389)
0808 04:51:01.675634 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.675651 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.675673 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:51:01.675739 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.675744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.675946 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.676103 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:01.676551 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.677747 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.680337 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.682027 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.682228 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.682389 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.682442 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.682523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":154,"threads_started":1}
I20260808 04:51:01.684202 28279 tablet_service.cc:1467] Processing CreateTablet for tablet c2c18f3a1e794cf6a4a3aaec1d15378f (DEFAULT_TABLE table=test-table [id=789723e6e281495a97f9c765ffb473b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:01.685470 28279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c18f3a1e794cf6a4a3aaec1d15378f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:01.685328 28063 tablet_service.cc:1467] Processing CreateTablet for tablet c2c18f3a1e794cf6a4a3aaec1d15378f (DEFAULT_TABLE table=test-table [id=789723e6e281495a97f9c765ffb473b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:01.685525 27860 tablet_service.cc:1467] Processing CreateTablet for tablet c2c18f3a1e794cf6a4a3aaec1d15378f (DEFAULT_TABLE table=test-table [id=789723e6e281495a97f9c765ffb473b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:01.686271 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c18f3a1e794cf6a4a3aaec1d15378f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:01.686436 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c18f3a1e794cf6a4a3aaec1d15378f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:01.693133 28420 tablet_bootstrap.cc:492] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Bootstrap starting.
I20260808 04:51:01.694772 28421 tablet_bootstrap.cc:492] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Bootstrap starting.
I20260808 04:51:01.695631 28420 tablet_bootstrap.cc:654] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:01.697099 28421 tablet_bootstrap.cc:654] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:01.696832 28420 log.cc:824] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:01.697813 28423 tablet_bootstrap.cc:492] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Bootstrap starting.
I20260808 04:51:01.697948 28421 log.cc:824] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:01.699847 28423 tablet_bootstrap.cc:654] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:01.699986 28421 tablet_bootstrap.cc:492] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: No bootstrap required, opened a new log
I20260808 04:51:01.700201 28421 ts_tablet_manager.cc:1397] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:51:01.700680 28423 log.cc:824] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:01.701601 28420 tablet_bootstrap.cc:492] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: No bootstrap required, opened a new log
I20260808 04:51:01.701822 28420 ts_tablet_manager.cc:1397] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260808 04:51:01.702702 28423 tablet_bootstrap.cc:492] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: No bootstrap required, opened a new log
I20260808 04:51:01.702888 28423 ts_tablet_manager.cc:1397] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:51:01.704604 28421 raft_consensus.cc:348] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.704885 28421 raft_consensus.cc:374] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:01.704954 28421 raft_consensus.cc:729] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16770a1d4516461bb4a6ec62ace82a18, State: Initialized, Role: FOLLOWER
I20260808 04:51:01.705420 28421 consensus_queue.cc:260] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.706087 28384 heartbeater.cc:503] Master 127.15.30.126:33285 was elected leader, sending a full tablet report...
I20260808 04:51:01.705966 28420 raft_consensus.cc:348] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.706259 28420 raft_consensus.cc:374] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:01.706314 28420 raft_consensus.cc:729] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c213451a804d5aa332b236ff079df7, State: Initialized, Role: FOLLOWER
I20260808 04:51:01.706437 28421 ts_tablet_manager.cc:1428] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260808 04:51:01.706707 28420 consensus_queue.cc:260] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.706799 28423 raft_consensus.cc:348] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.707008 28423 raft_consensus.cc:374] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:01.707067 28423 raft_consensus.cc:729] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f9a8a059d84f6a9b8b40c559632c12, State: Initialized, Role: FOLLOWER
I20260808 04:51:01.707338 27960 heartbeater.cc:503] Master 127.15.30.126:33285 was elected leader, sending a full tablet report...
I20260808 04:51:01.707633 28420 ts_tablet_manager.cc:1428] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:51:01.707492 28423 consensus_queue.cc:260] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.709563 28174 heartbeater.cc:503] Master 127.15.30.126:33285 was elected leader, sending a full tablet report...
I20260808 04:51:01.709872 28423 ts_tablet_manager.cc:1428] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260808 04:51:01.855394 28385 tablet.cc:2369] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:51:01.869017 28175 tablet.cc:2369] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:51:01.879972 28436 raft_consensus.cc:482] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:51:01.880179 28436 raft_consensus.cc:504] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.881289 28436 leader_election.cc:288] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685), c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829)
I20260808 04:51:01.885345 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.881732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:52073 (local address 127.15.30.67:45685)
0808 04:51:01.881908 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.881913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.882168 (+ 255us) server_negotiation.cc:407] Connection header received
0808 04:51:01.882331 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.882334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.882378 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.882453 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:01.882742 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.883248 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.884394 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.884794 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.884920 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.884972 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.885152 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.885219 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:51:01.885351 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.881648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:45685 (local address 127.15.30.66:52073)
0808 04:51:01.882068 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.882080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.882225 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.882475 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.882478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.882494 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:01.882620 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.882625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.883357 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.883361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.884240 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.884250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.884916 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.884920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.885007 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.885049 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.885136 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.885212 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":23,"thread_start_us":68,"threads_started":1}
I20260808 04:51:01.885423 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.881724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:39421 (local address 127.15.30.65:46829)
0808 04:51:01.881867 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.881872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.882044 (+ 172us) server_negotiation.cc:407] Connection header received
0808 04:51:01.882221 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.882230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.882272 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.882359 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:01.882648 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.883188 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.884409 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.884817 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.884946 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.884996 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.885226 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.885294 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:51:01.885814 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.881548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46829 (local address 127.15.30.66:39421)
0808 04:51:01.881921 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:01.881937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:01.882089 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:01.882374 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:01.882377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:01.882394 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:01.882523 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.882528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.883308 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.883311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.884252 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:01.884262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.884946 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.884950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:01.885098 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.885116 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:51:01.885210 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:51:01.885284 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":103,"thread_start_us":113,"threads_started":1}
I20260808 04:51:01.885936 28308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c18f3a1e794cf6a4a3aaec1d15378f" candidate_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16770a1d4516461bb4a6ec62ace82a18" is_pre_election: true
I20260808 04:51:01.886258 28308 raft_consensus.cc:2454] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f9a8a059d84f6a9b8b40c559632c12 in term 0.
I20260808 04:51:01.886559 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c18f3a1e794cf6a4a3aaec1d15378f" candidate_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c213451a804d5aa332b236ff079df7" is_pre_election: true
I20260808 04:51:01.886780 27999 leader_election.cc:302] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, a3f9a8a059d84f6a9b8b40c559632c12; no voters:
I20260808 04:51:01.886852 27891 raft_consensus.cc:2454] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f9a8a059d84f6a9b8b40c559632c12 in term 0.
I20260808 04:51:01.887066 28436 raft_consensus.cc:2781] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:51:01.887202 28436 raft_consensus.cc:482] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:51:01.887267 28436 raft_consensus.cc:3037] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:01.888324 28436 raft_consensus.cc:504] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
W20260808 04:51:01.888685 27961 tablet.cc:2369] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:51:01.888749 28436 leader_election.cc:288] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [CANDIDATE]: Term 1 election: Requested vote from peers 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685), c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829)
I20260808 04:51:01.889111 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c18f3a1e794cf6a4a3aaec1d15378f" candidate_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c213451a804d5aa332b236ff079df7"
I20260808 04:51:01.889242 27891 raft_consensus.cc:3037] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:01.890264 27891 raft_consensus.cc:2454] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f9a8a059d84f6a9b8b40c559632c12 in term 1.
I20260808 04:51:01.890569 27997 leader_election.cc:302] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3f9a8a059d84f6a9b8b40c559632c12, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:01.890543 28308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c18f3a1e794cf6a4a3aaec1d15378f" candidate_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16770a1d4516461bb4a6ec62ace82a18"
I20260808 04:51:01.890702 28308 raft_consensus.cc:3037] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:01.890764 28436 raft_consensus.cc:2781] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:01.890975 28436 raft_consensus.cc:686] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 LEADER]: Becoming Leader. State: Replica: a3f9a8a059d84f6a9b8b40c559632c12, State: Running, Role: LEADER
I20260808 04:51:01.891312 28436 consensus_queue.cc:237] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } }
I20260808 04:51:01.892086 28308 raft_consensus.cc:2454] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f9a8a059d84f6a9b8b40c559632c12 in term 1.
I20260808 04:51:01.894546 27650 catalog_manager.cc:5495] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66). New cstate: current_term: 1 leader_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } health_report { overall_health: UNKNOWN } } }
I20260808 04:51:01.915369 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:01.911752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.15.30.66:42389)
0808 04:51:01.911865 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:01.911870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:01.911884 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:51:01.912013 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:01.912015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:01.912075 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:01.912229 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:01.912463 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.912950 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.913817 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:01.914079 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:01.914191 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:01.914460 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:01.914877 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:01.914995 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:01.915071 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 04:51:01.915175 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 04:51:01.920754 28308 raft_consensus.cc:1261] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Refusing update from remote peer a3f9a8a059d84f6a9b8b40c559632c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:01.920771 27891 raft_consensus.cc:1261] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Refusing update from remote peer a3f9a8a059d84f6a9b8b40c559632c12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:01.921427 28436 consensus_queue.cc:1035] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:01.921633 28439 consensus_queue.cc:1035] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:01.929423 28446 mvcc.cc:205] Tried to move back new op lower bound from 7316130455221018624 to 7316130455109627904. Current Snapshot: MvccSnapshot[applied={T|T < 7316130455221018624}]
I20260808 04:51:01.937844 27650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:01.938037 27650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:01.940883 27650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Sending DeleteTablet for 3 replicas of tablet c2c18f3a1e794cf6a4a3aaec1d15378f
I20260808 04:51:01.941668 28063 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c18f3a1e794cf6a4a3aaec1d15378f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:01 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:38762
I20260808 04:51:01.941855 27860 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c18f3a1e794cf6a4a3aaec1d15378f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:01 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39694
I20260808 04:51:01.942189 28279 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c18f3a1e794cf6a4a3aaec1d15378f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:01 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39584
I20260808 04:51:01.942406 28461 tablet_replica.cc:330] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: stopping tablet replica
I20260808 04:51:01.942682 28460 tablet_replica.cc:330] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: stopping tablet replica
I20260808 04:51:01.942873 28461 raft_consensus.cc:2229] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:01.942953 28460 raft_consensus.cc:2229] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:51:01.943121 27650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:51:01.943370 28460 raft_consensus.cc:2258] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:01.943486 28461 raft_consensus.cc:2258] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 04:51:01.943521 27650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:51:01.943929 28462 tablet_replica.cc:330] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: stopping tablet replica
I20260808 04:51:01.944293 28462 raft_consensus.cc:2229] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:01.944696 28462 raft_consensus.cc:2258] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:01.945194 28461 ts_tablet_manager.cc:1905] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:01.946206 28462 ts_tablet_manager.cc:1905] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:01.947937 28461 ts_tablet_manager.cc:1918] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:01.948043 28461 log.cc:1196] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/c2c18f3a1e794cf6a4a3aaec1d15378f
I20260808 04:51:01.948428 28461 ts_tablet_manager.cc:1939] T c2c18f3a1e794cf6a4a3aaec1d15378f P c8c213451a804d5aa332b236ff079df7: Deleting consensus metadata
I20260808 04:51:01.948500 28279 tablet_service.cc:1467] Processing CreateTablet for tablet eca5c3da324b42b08089bdd0737f8b90 (DEFAULT_TABLE table=test-table [id=e5d270c4689c46e09db36d4e875a2fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:01.948781 28279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca5c3da324b42b08089bdd0737f8b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:01.949437 28462 ts_tablet_manager.cc:1918] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:01.949606 28462 log.cc:1196] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/c2c18f3a1e794cf6a4a3aaec1d15378f
I20260808 04:51:01.950018 28462 ts_tablet_manager.cc:1939] T c2c18f3a1e794cf6a4a3aaec1d15378f P 16770a1d4516461bb4a6ec62ace82a18: Deleting consensus metadata
I20260808 04:51:01.950580 28460 ts_tablet_manager.cc:1905] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:01.950951 28421 tablet_bootstrap.cc:492] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: Bootstrap starting.
I20260808 04:51:01.951349 27632 catalog_manager.cc:4841] TS 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685): tablet c2c18f3a1e794cf6a4a3aaec1d15378f (table test-table [id=789723e6e281495a97f9c765ffb473b3]) successfully deleted
I20260808 04:51:01.951817 27631 catalog_manager.cc:4841] TS c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829): tablet c2c18f3a1e794cf6a4a3aaec1d15378f (table test-table [id=789723e6e281495a97f9c765ffb473b3]) successfully deleted
I20260808 04:51:01.952005 28421 tablet_bootstrap.cc:654] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:01.952196 27860 tablet_service.cc:1467] Processing CreateTablet for tablet eca5c3da324b42b08089bdd0737f8b90 (DEFAULT_TABLE table=test-table [id=e5d270c4689c46e09db36d4e875a2fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:01.952486 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca5c3da324b42b08089bdd0737f8b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:01.953099 28460 ts_tablet_manager.cc:1918] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:01.953218 28460 log.cc:1196] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/c2c18f3a1e794cf6a4a3aaec1d15378f
I20260808 04:51:01.953544 28460 ts_tablet_manager.cc:1939] T c2c18f3a1e794cf6a4a3aaec1d15378f P a3f9a8a059d84f6a9b8b40c559632c12: Deleting consensus metadata
I20260808 04:51:01.954576 28420 tablet_bootstrap.cc:492] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: Bootstrap starting.
I20260808 04:51:01.954887 27633 catalog_manager.cc:4841] TS a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389): tablet c2c18f3a1e794cf6a4a3aaec1d15378f (table test-table [id=789723e6e281495a97f9c765ffb473b3]) successfully deleted
I20260808 04:51:01.955201 28063 tablet_service.cc:1467] Processing CreateTablet for tablet eca5c3da324b42b08089bdd0737f8b90 (DEFAULT_TABLE table=test-table [id=e5d270c4689c46e09db36d4e875a2fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:01.955431 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca5c3da324b42b08089bdd0737f8b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:01.955549 28420 tablet_bootstrap.cc:654] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:01.956859 28421 tablet_bootstrap.cc:492] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: No bootstrap required, opened a new log
I20260808 04:51:01.956952 28421 ts_tablet_manager.cc:1397] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260808 04:51:01.957264 28420 tablet_bootstrap.cc:492] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: No bootstrap required, opened a new log
I20260808 04:51:01.957358 28420 ts_tablet_manager.cc:1397] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:51:01.957443 28421 raft_consensus.cc:348] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:01.957556 28421 raft_consensus.cc:374] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:01.957598 28421 raft_consensus.cc:729] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16770a1d4516461bb4a6ec62ace82a18, State: Initialized, Role: FOLLOWER
I20260808 04:51:01.957726 28421 consensus_queue.cc:260] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:01.957875 28423 tablet_bootstrap.cc:492] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: Bootstrap starting.
I20260808 04:51:01.957811 28420 raft_consensus.cc:348] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:01.957922 28420 raft_consensus.cc:374] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:01.957953 28420 raft_consensus.cc:729] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c213451a804d5aa332b236ff079df7, State: Initialized, Role: FOLLOWER
I20260808 04:51:01.958022 28421 ts_tablet_manager.cc:1428] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:51:01.958060 28420 consensus_queue.cc:260] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:01.958384 28420 ts_tablet_manager.cc:1428] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:01.958825 28423 tablet_bootstrap.cc:654] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:01.960398 28423 tablet_bootstrap.cc:492] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: No bootstrap required, opened a new log
I20260808 04:51:01.960489 28423 ts_tablet_manager.cc:1397] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:51:01.961141 28423 raft_consensus.cc:348] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:01.961273 28423 raft_consensus.cc:374] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:01.961335 28423 raft_consensus.cc:729] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f9a8a059d84f6a9b8b40c559632c12, State: Initialized, Role: FOLLOWER
I20260808 04:51:01.961459 28423 consensus_queue.cc:260] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:01.961732 28423 ts_tablet_manager.cc:1428] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:02.051755 28432 raft_consensus.cc:482] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:51:02.051946 28432 raft_consensus.cc:504] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.053220 28432 leader_election.cc:288] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829), a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389)
I20260808 04:51:02.057847 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.053401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46829 (local address 127.15.30.67:50445)
0808 04:51:02.053629 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.053647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.053749 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.054249 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.054252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.054265 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:02.054409 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.054415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.055408 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.055412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.056320 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.056330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.057369 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.057373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.057481 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.057503 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.057588 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.057679 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260808 04:51:02.057897 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.053831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:50445 (local address 127.15.30.65:46829)
0808 04:51:02.053991 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.053997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.054014 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:51:02.054080 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.054084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.054138 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.054214 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:02.054546 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.055278 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.056493 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.057146 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.057260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.057311 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.057648 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.057766 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:51:02.058378 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca5c3da324b42b08089bdd0737f8b90" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c213451a804d5aa332b236ff079df7" is_pre_election: true
I20260808 04:51:02.058570 27891 raft_consensus.cc:2454] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 0.
I20260808 04:51:02.058702 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.053727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:41769 (local address 127.15.30.66:42389)
0808 04:51:02.053867 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.053872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.054128 (+ 256us) server_negotiation.cc:407] Connection header received
0808 04:51:02.054225 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.054229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.054288 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.054399 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:02.054706 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.055461 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.057009 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.057589 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.057710 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.057758 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.058484 (+ 726us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.058569 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:51:02.058816 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.053495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:42389 (local address 127.15.30.67:41769)
0808 04:51:02.053987 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.054000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.054129 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.054409 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.054412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.054428 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:02.054566 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.054574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.055610 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.055613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.056893 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.056905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.058205 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.058212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.058332 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.058356 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.058440 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.058595 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":79,"threads_started":1}
I20260808 04:51:02.059262 28204 leader_election.cc:302] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:02.059398 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca5c3da324b42b08089bdd0737f8b90" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" is_pre_election: true
I20260808 04:51:02.059597 28432 raft_consensus.cc:2781] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:51:02.059629 28097 raft_consensus.cc:2454] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 0.
I20260808 04:51:02.059667 28432 raft_consensus.cc:482] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:51:02.059710 28432 raft_consensus.cc:3037] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.061133 28432 raft_consensus.cc:504] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.061539 28432 leader_election.cc:288] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 election: Requested vote from peers c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829), a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389)
I20260808 04:51:02.061897 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca5c3da324b42b08089bdd0737f8b90" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c213451a804d5aa332b236ff079df7"
I20260808 04:51:02.062021 27891 raft_consensus.cc:3037] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.062955 27891 raft_consensus.cc:2454] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 1.
I20260808 04:51:02.063227 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca5c3da324b42b08089bdd0737f8b90" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12"
I20260808 04:51:02.063274 28204 leader_election.cc:302] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:02.063361 28097 raft_consensus.cc:3037] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.063443 28432 raft_consensus.cc:2781] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:02.063640 28432 raft_consensus.cc:686] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 1 LEADER]: Becoming Leader. State: Replica: 16770a1d4516461bb4a6ec62ace82a18, State: Running, Role: LEADER
I20260808 04:51:02.063928 28432 consensus_queue.cc:237] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.064885 28097 raft_consensus.cc:2454] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 1.
I20260808 04:51:02.066455 27652 catalog_manager.cc:5495] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67). New cstate: current_term: 1 leader_uuid: "16770a1d4516461bb4a6ec62ace82a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } health_report { overall_health: HEALTHY } } }
Aug 08 04:51:02 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164660, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.126@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:51:02.111903 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.098614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.15.30.126:33285)
0808 04:51:02.098765 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.098770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.099127 (+ 357us) server_negotiation.cc:407] Connection header received
0808 04:51:02.099283 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.099287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.099347 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.099421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:02.103827 (+ 4406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.104353 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.105704 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.106078 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.106184 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.108769 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:02.108772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:02.108774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:02.108795 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:02.110250 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:02.110727 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:02.110730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:02.110732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:02.110812 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:02.111075 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:02.111077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:02.111078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:02.111256 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:02.111334 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.111684 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.111787 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:51:02.111908 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.098722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33285 (local address 127.0.0.1:56348)
0808 04:51:02.099046 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.099061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.099192 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.099461 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.099464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.100006 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:02.100151 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.100161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.104523 (+ 4362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.104529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.105556 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.105571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.106214 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.106219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.106312 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.107048 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:02.107061 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:02.108625 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:02.110385 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:02.110395 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:02.110401 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:02.110617 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:02.110900 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:02.110902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:02.110904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:02.110982 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:02.111351 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:02.111357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:02.111501 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.111599 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.111715 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260808 04:51:02.116086 27652 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:56348:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:51:02.116900 27652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:51:02.122114 28063 tablet_service.cc:1467] Processing CreateTablet for tablet a9e6d42952984be9bd35991f9fd6bde8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1cbee37ebb84152891e703ec88802f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:51:02.122116 27860 tablet_service.cc:1467] Processing CreateTablet for tablet a9e6d42952984be9bd35991f9fd6bde8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1cbee37ebb84152891e703ec88802f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:51:02.122165 28279 tablet_service.cc:1467] Processing CreateTablet for tablet a9e6d42952984be9bd35991f9fd6bde8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1cbee37ebb84152891e703ec88802f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:51:02.122592 28279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e6d42952984be9bd35991f9fd6bde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.122592 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e6d42952984be9bd35991f9fd6bde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.122612 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e6d42952984be9bd35991f9fd6bde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.125319 28420 tablet_bootstrap.cc:492] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: Bootstrap starting.
I20260808 04:51:02.125658 28423 tablet_bootstrap.cc:492] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: Bootstrap starting.
I20260808 04:51:02.125679 28421 tablet_bootstrap.cc:492] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: Bootstrap starting.
I20260808 04:51:02.126329 28420 tablet_bootstrap.cc:654] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.126449 28423 tablet_bootstrap.cc:654] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.126662 28421 tablet_bootstrap.cc:654] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.127681 28420 tablet_bootstrap.cc:492] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: No bootstrap required, opened a new log
I20260808 04:51:02.127779 28420 ts_tablet_manager.cc:1397] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:51:02.128090 28421 tablet_bootstrap.cc:492] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: No bootstrap required, opened a new log
I20260808 04:51:02.128186 28421 ts_tablet_manager.cc:1397] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 04:51:02.128293 28423 tablet_bootstrap.cc:492] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: No bootstrap required, opened a new log
I20260808 04:51:02.128265 28420 raft_consensus.cc:348] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.128407 28420 raft_consensus.cc:374] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.128451 28420 raft_consensus.cc:729] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c213451a804d5aa332b236ff079df7, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.128578 28423 ts_tablet_manager.cc:1397] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:51:02.128572 28420 consensus_queue.cc:260] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.128681 28421 raft_consensus.cc:348] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.128782 28421 raft_consensus.cc:374] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.128816 28421 raft_consensus.cc:729] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16770a1d4516461bb4a6ec62ace82a18, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.128903 28420 ts_tablet_manager.cc:1428] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:02.128861 28435 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.128937 28421 consensus_queue.cc:260] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.129020 28435 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:51:02.129076 28423 raft_consensus.cc:348] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.129189 28423 raft_consensus.cc:374] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.129220 28423 raft_consensus.cc:729] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f9a8a059d84f6a9b8b40c559632c12, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.129220 28421 ts_tablet_manager.cc:1428] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:02.129344 28423 consensus_queue.cc:260] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.129513 28432 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.129626 28423 ts_tablet_manager.cc:1428] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:02.129647 28432 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:51:02.129818 28439 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.129941 28439 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:51:02.282553 28435 raft_consensus.cc:482] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:51:02.282773 28435 raft_consensus.cc:504] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.284065 28435 leader_election.cc:288] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389), 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685)
I20260808 04:51:02.287945 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.284554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:51015 (local address 127.15.30.67:45685)
0808 04:51:02.284683 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.284690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.284850 (+ 160us) server_negotiation.cc:407] Connection header received
0808 04:51:02.285008 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.285011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.285062 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.285158 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:02.285401 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.285893 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.286982 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.287391 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.287524 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.287579 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.287762 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.287834 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:51:02.287961 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.284353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:45685 (local address 127.15.30.65:51015)
0808 04:51:02.284777 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.284790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.284885 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.285170 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.285173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.285189 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:02.285290 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.285295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.286032 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.286036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.286854 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.286863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.287520 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.287523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.287617 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.287646 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.287731 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.287801 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260808 04:51:02.288080 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.284359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:51609 (local address 127.15.30.66:42389)
0808 04:51:02.284552 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.284559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.284784 (+ 225us) server_negotiation.cc:407] Connection header received
0808 04:51:02.284937 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.284940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.284993 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.285078 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:02.285329 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.285849 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.287157 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.287533 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.287664 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.287712 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.287884 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.287956 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 04:51:02.288110 28490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.284261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:42389 (local address 127.15.30.65:51609)
0808 04:51:02.284680 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.284694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.284837 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.285094 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.285097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.285116 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:02.285225 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.285230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.285972 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.285976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.286851 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.286860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.287665 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.287668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.287752 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.287778 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.287857 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.287924 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":25,"thread_start_us":147,"threads_started":1}
I20260808 04:51:02.288542 28308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e6d42952984be9bd35991f9fd6bde8" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16770a1d4516461bb4a6ec62ace82a18" is_pre_election: true
I20260808 04:51:02.288713 28308 raft_consensus.cc:2454] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 0.
I20260808 04:51:02.288766 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e6d42952984be9bd35991f9fd6bde8" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" is_pre_election: true
I20260808 04:51:02.288902 28097 raft_consensus.cc:2454] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 0.
I20260808 04:51:02.289122 27794 leader_election.cc:302] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:02.289379 28435 raft_consensus.cc:2781] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:51:02.289455 28435 raft_consensus.cc:482] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:51:02.289511 28435 raft_consensus.cc:3037] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.290483 28435 raft_consensus.cc:504] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.290820 28435 leader_election.cc:288] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 election: Requested vote from peers a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389), 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685)
I20260808 04:51:02.291190 28308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e6d42952984be9bd35991f9fd6bde8" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16770a1d4516461bb4a6ec62ace82a18"
I20260808 04:51:02.291256 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e6d42952984be9bd35991f9fd6bde8" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12"
I20260808 04:51:02.291313 28308 raft_consensus.cc:3037] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.291361 28097 raft_consensus.cc:3037] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.292503 28097 raft_consensus.cc:2454] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 1.
I20260808 04:51:02.292503 28308 raft_consensus.cc:2454] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 1.
I20260808 04:51:02.293231 27796 leader_election.cc:302] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3f9a8a059d84f6a9b8b40c559632c12, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:02.293407 28435 raft_consensus.cc:2781] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:02.293607 28435 raft_consensus.cc:686] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [term 1 LEADER]: Becoming Leader. State: Replica: c8c213451a804d5aa332b236ff079df7, State: Running, Role: LEADER
I20260808 04:51:02.293875 28435 consensus_queue.cc:237] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.295116 28492 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: TxnStatusTablet state changed. Reason: New leader c8c213451a804d5aa332b236ff079df7. Latest consensus state: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.295251 28492 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:51:02.295522 28494 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:51:02.295708 28494 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:51:02.296482 27650 catalog_manager.cc:5495] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8c213451a804d5aa332b236ff079df7 (127.15.30.65). New cstate: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } health_report { overall_health: UNKNOWN } } }
I20260808 04:51:02.319445 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.315388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46829 (local address 127.0.0.1:39708)
0808 04:51:02.315520 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.315537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.315642 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.315990 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.315992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.315998 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:51:02.316087 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.316091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.317115 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.317118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.317732 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.317741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.318322 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.318324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.318387 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.318432 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.319193 (+ 761us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.319209 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.319269 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.319335 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 04:51:02.319465 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.315464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.15.30.65:46829)
0808 04:51:02.315584 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.315588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.315605 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:51:02.315783 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.315786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.315868 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.315976 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:02.316188 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.316993 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.317847 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.318210 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.318318 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.318571 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.319070 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.319193 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.319282 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.319338 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:51:02.322609 28308 raft_consensus.cc:1261] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Refusing update from remote peer c8c213451a804d5aa332b236ff079df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:02.322770 28097 raft_consensus.cc:1261] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Refusing update from remote peer c8c213451a804d5aa332b236ff079df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:02.323144 28492 consensus_queue.cc:1035] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:02.323365 28435 consensus_queue.cc:1035] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:02.324891 28432 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: TxnStatusTablet state changed. Reason: New leader c8c213451a804d5aa332b236ff079df7. Latest consensus state: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.325002 28432 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:51:02.325047 28439 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: TxnStatusTablet state changed. Reason: New leader c8c213451a804d5aa332b236ff079df7. Latest consensus state: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.325158 28439 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:51:02.327450 28435 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.327556 28435 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:51:02.328518 28444 mvcc.cc:205] Tried to move back new op lower bound from 7316130456868290560 to 7316130456758353920. Current Snapshot: MvccSnapshot[applied={T|T < 7316130456868290560}]
I20260808 04:51:02.328856 28439 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.328958 28439 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P a3f9a8a059d84f6a9b8b40c559632c12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:51:02.331507 28435 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.331621 28435 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P c8c213451a804d5aa332b236ff079df7: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:51:02.334187 28432 tablet_replica.cc:439] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } } }
I20260808 04:51:02.334311 28432 tablet_replica.cc:442] T a9e6d42952984be9bd35991f9fd6bde8 P 16770a1d4516461bb4a6ec62ace82a18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:51:02.334416 28447 mvcc.cc:205] Tried to move back new op lower bound from 7316130456868290560 to 7316130456758353920. Current Snapshot: MvccSnapshot[applied={T|T < 7316130456868290560 or (T in {7316130456868290560})}]
I20260808 04:51:02.336302 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.331920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.15.30.67:45685)
0808 04:51:02.332100 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.332106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.332121 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:51:02.332204 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.332207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.332273 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.332602 (+ 329us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:02.332706 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.333231 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.334564 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.334901 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.335008 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.335316 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.335844 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.335979 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.336100 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.336171 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 04:51:02.347348 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.343227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:45087 (local address 127.15.30.65:46829)
0808 04:51:02.343338 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.343342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.343466 (+ 124us) server_negotiation.cc:407] Connection header received
0808 04:51:02.343573 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.343575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.343613 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.343682 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:02.343946 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.344514 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.346373 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.346574 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.346656 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.346971 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.347031 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.347079 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.347217 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.347256 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:51:02.347438 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.343060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46829 (local address 127.15.30.67:45087)
0808 04:51:02.343376 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.343391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.343493 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.343720 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.343723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.343731 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:51:02.343843 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.343848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.344626 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.344630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.346245 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.346257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.346685 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.346688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.346784 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.346834 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.347117 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.347124 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.347193 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.347272 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
I20260808 04:51:02.354161 27891 raft_consensus.cc:1261] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Refusing update from remote peer 16770a1d4516461bb4a6ec62ace82a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:02.354508 28097 raft_consensus.cc:1261] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Refusing update from remote peer 16770a1d4516461bb4a6ec62ace82a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:02.354627 28470 consensus_queue.cc:1035] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:02.354938 28432 consensus_queue.cc:1035] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:02.381629 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.377647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:36627 (local address 127.15.30.67:45685)
0808 04:51:02.377752 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.377756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.377770 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:51:02.377866 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.377868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.377916 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.377995 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:02.378292 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.379056 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.380357 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.380727 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.380835 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.381140 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.381219 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.381295 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.381429 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.381486 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:51:02.381778 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.377289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:45685 (local address 127.15.30.65:36627)
0808 04:51:02.377638 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:02.377654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:02.377753 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:02.378020 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:02.378023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:02.378031 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:51:02.378166 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.378172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.379196 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.379199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.380224 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:02.380234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.380846 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.380850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:02.380949 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.381000 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.381313 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.381319 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:51:02.381380 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 04:51:02.381455 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":128,"threads_started":1}
I20260808 04:51:02.413280 27650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:02.413460 27650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:02.415934 27650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Sending DeleteTablet for 3 replicas of tablet eca5c3da324b42b08089bdd0737f8b90
I20260808 04:51:02.416554 28279 tablet_service.cc:1514] Processing DeleteTablet for tablet eca5c3da324b42b08089bdd0737f8b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:02 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39584
I20260808 04:51:02.416734 28462 tablet_replica.cc:330] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: stopping tablet replica
I20260808 04:51:02.416734 28063 tablet_service.cc:1514] Processing DeleteTablet for tablet eca5c3da324b42b08089bdd0737f8b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:02 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:38762
I20260808 04:51:02.416885 28462 raft_consensus.cc:2229] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:51:02.416909 28460 tablet_replica.cc:330] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: stopping tablet replica
I20260808 04:51:02.417006 28460 raft_consensus.cc:2229] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:02.417131 28460 raft_consensus.cc:2258] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:02.417176 28462 raft_consensus.cc:2258] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:02.417944 28460 ts_tablet_manager.cc:1905] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:02.417965 28462 ts_tablet_manager.cc:1905] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:02.422386 28462 ts_tablet_manager.cc:1918] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:51:02.422557 28462 log.cc:1196] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/eca5c3da324b42b08089bdd0737f8b90
I20260808 04:51:02.422902 28462 ts_tablet_manager.cc:1939] T eca5c3da324b42b08089bdd0737f8b90 P 16770a1d4516461bb4a6ec62ace82a18: Deleting consensus metadata
I20260808 04:51:02.423650 27632 catalog_manager.cc:4841] TS 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685): tablet eca5c3da324b42b08089bdd0737f8b90 (table test-table [id=e5d270c4689c46e09db36d4e875a2fee]) successfully deleted
I20260808 04:51:02.423890 28460 ts_tablet_manager.cc:1918] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:51:02.423970 28460 log.cc:1196] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/eca5c3da324b42b08089bdd0737f8b90
I20260808 04:51:02.424235 28460 ts_tablet_manager.cc:1939] T eca5c3da324b42b08089bdd0737f8b90 P a3f9a8a059d84f6a9b8b40c559632c12: Deleting consensus metadata
I20260808 04:51:02.426558 27860 tablet_service.cc:1514] Processing DeleteTablet for tablet eca5c3da324b42b08089bdd0737f8b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:02 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39694
I20260808 04:51:02.426810 28461 tablet_replica.cc:330] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: stopping tablet replica
I20260808 04:51:02.426921 28461 raft_consensus.cc:2229] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:02.427038 28461 raft_consensus.cc:2258] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:02.427832 28461 ts_tablet_manager.cc:1905] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:02.430636 28461 ts_tablet_manager.cc:1918] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:51:02.430722 28461 log.cc:1196] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/eca5c3da324b42b08089bdd0737f8b90
I20260808 04:51:02.430980 28461 ts_tablet_manager.cc:1939] T eca5c3da324b42b08089bdd0737f8b90 P c8c213451a804d5aa332b236ff079df7: Deleting consensus metadata
I20260808 04:51:02.431733 27631 catalog_manager.cc:4841] TS c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829): tablet eca5c3da324b42b08089bdd0737f8b90 (table test-table [id=e5d270c4689c46e09db36d4e875a2fee]) successfully deleted
I20260808 04:51:02.431809 27633 catalog_manager.cc:4841] TS a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389): tablet eca5c3da324b42b08089bdd0737f8b90 (table test-table [id=e5d270c4689c46e09db36d4e875a2fee]) successfully deleted
Aug 08 04:51:02 dist-test-slave-zpvx krb5kdc[27592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:51:02 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:02 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:51:02.460879 27724 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:02.453012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.15.30.126:33285)
0808 04:51:02.453170 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.453174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.453443 (+ 269us) server_negotiation.cc:407] Connection header received
0808 04:51:02.453582 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.453585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.453645 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.453747 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:02.454618 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.455446 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.456733 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.457054 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.457161 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.460746 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56360: BlockingRecv error: recv got EOF from 127.0.0.1:56360 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 04:51:02 dist-test-slave-zpvx krb5kdc[27592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM
I20260808 04:51:02.473424 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.462860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.15.30.126:33285)
0808 04:51:02.462969 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.462973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.463379 (+ 406us) server_negotiation.cc:407] Connection header received
0808 04:51:02.463499 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.463502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.463549 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.463621 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:02.464328 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.465104 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.466240 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.466570 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.466672 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.469463 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:02.469467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:02.469470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:02.469497 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:02.471290 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:02.471806 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:02.471810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:02.471812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:02.471906 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:02.472284 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:02.472290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:02.472293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:02.472555 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:02.472752 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.472991 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.473113 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:51:02.476213 27650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:51:02.476711 27650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:51:02.481467 27860 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac (DEFAULT_TABLE table=test-table [id=f5c902312acb45e29e182ebfe2d0c5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:02.481776 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.481889 28279 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac (DEFAULT_TABLE table=test-table [id=f5c902312acb45e29e182ebfe2d0c5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:02.482210 28279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.483237 28063 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac (DEFAULT_TABLE table=test-table [id=f5c902312acb45e29e182ebfe2d0c5e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:02.483522 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.484831 28421 tablet_bootstrap.cc:492] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: Bootstrap starting.
I20260808 04:51:02.485661 28421 tablet_bootstrap.cc:654] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.485940 28423 tablet_bootstrap.cc:492] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: Bootstrap starting.
I20260808 04:51:02.486716 28423 tablet_bootstrap.cc:654] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.487813 28420 tablet_bootstrap.cc:492] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: Bootstrap starting.
I20260808 04:51:02.488638 28420 tablet_bootstrap.cc:654] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.489375 28421 tablet_bootstrap.cc:492] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: No bootstrap required, opened a new log
I20260808 04:51:02.489470 28421 ts_tablet_manager.cc:1397] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 04:51:02.489624 28423 tablet_bootstrap.cc:492] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: No bootstrap required, opened a new log
I20260808 04:51:02.489730 28423 ts_tablet_manager.cc:1397] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 04:51:02.489939 28421 raft_consensus.cc:348] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.490054 28421 raft_consensus.cc:374] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.490085 28421 raft_consensus.cc:729] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16770a1d4516461bb4a6ec62ace82a18, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.490201 28421 consensus_queue.cc:260] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.490188 28423 raft_consensus.cc:348] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.490295 28423 raft_consensus.cc:374] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.490329 28423 raft_consensus.cc:729] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f9a8a059d84f6a9b8b40c559632c12, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.490365 28420 tablet_bootstrap.cc:492] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: No bootstrap required, opened a new log
I20260808 04:51:02.490437 28420 ts_tablet_manager.cc:1397] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 04:51:02.490447 28421 ts_tablet_manager.cc:1428] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:02.490449 28423 consensus_queue.cc:260] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.490713 28423 ts_tablet_manager.cc:1428] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:02.490832 28420 raft_consensus.cc:348] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.490939 28420 raft_consensus.cc:374] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.490971 28420 raft_consensus.cc:729] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c213451a804d5aa332b236ff079df7, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.491083 28420 consensus_queue.cc:260] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.491367 28420 ts_tablet_manager.cc:1428] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:51:02.687400 28435 raft_consensus.cc:482] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:51:02.687587 28435 raft_consensus.cc:504] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.688167 28435 leader_election.cc:288] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389), 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685)
I20260808 04:51:02.688783 28308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e0f3df4bb4a09b2e2bfd328e236ac" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16770a1d4516461bb4a6ec62ace82a18" is_pre_election: true
I20260808 04:51:02.688786 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e0f3df4bb4a09b2e2bfd328e236ac" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" is_pre_election: true
I20260808 04:51:02.688951 28308 raft_consensus.cc:2454] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 0.
I20260808 04:51:02.688951 28097 raft_consensus.cc:2454] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 0.
I20260808 04:51:02.689271 27794 leader_election.cc:302] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:02.689519 28435 raft_consensus.cc:2781] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:51:02.689601 28435 raft_consensus.cc:482] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:51:02.689651 28435 raft_consensus.cc:3037] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.690915 28435 raft_consensus.cc:504] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.691309 28435 leader_election.cc:288] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 election: Requested vote from peers a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389), 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685)
I20260808 04:51:02.691781 28308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e0f3df4bb4a09b2e2bfd328e236ac" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16770a1d4516461bb4a6ec62ace82a18"
I20260808 04:51:02.691915 28308 raft_consensus.cc:3037] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.692559 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e0f3df4bb4a09b2e2bfd328e236ac" candidate_uuid: "c8c213451a804d5aa332b236ff079df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12"
I20260808 04:51:02.692710 28097 raft_consensus.cc:3037] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:02.693416 28308 raft_consensus.cc:2454] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 1.
I20260808 04:51:02.693804 27794 leader_election.cc:302] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:02.693974 28435 raft_consensus.cc:2781] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:02.694079 28435 raft_consensus.cc:686] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 1 LEADER]: Becoming Leader. State: Replica: c8c213451a804d5aa332b236ff079df7, State: Running, Role: LEADER
I20260808 04:51:02.694209 28097 raft_consensus.cc:2454] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8c213451a804d5aa332b236ff079df7 in term 1.
I20260808 04:51:02.694291 28435 consensus_queue.cc:237] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.695904 27649 catalog_manager.cc:5495] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8c213451a804d5aa332b236ff079df7 (127.15.30.65). New cstate: current_term: 1 leader_uuid: "c8c213451a804d5aa332b236ff079df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } health_report { overall_health: UNKNOWN } } }
I20260808 04:51:02.732656 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:02.729653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.15.30.65:46829)
0808 04:51:02.729800 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:02.729804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:02.729819 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:51:02.729866 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:02.729869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:02.729915 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:02.729990 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:02.730273 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.730834 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.731687 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:02.731912 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:02.731998 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:02.732246 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:02.732312 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:02.732432 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:02.732498 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 04:51:02.732543 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:51:02.735013 28097 raft_consensus.cc:1261] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Refusing update from remote peer c8c213451a804d5aa332b236ff079df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:02.734999 28308 raft_consensus.cc:1261] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Refusing update from remote peer c8c213451a804d5aa332b236ff079df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:02.735463 28492 consensus_queue.cc:1035] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:02.735644 28435 consensus_queue.cc:1035] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:02.744295 27649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:02.744472 27649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:02.746434 27649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Sending DeleteTablet for 3 replicas of tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac
I20260808 04:51:02.746956 27860 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:02 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39694
I20260808 04:51:02.747053 28279 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:02 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39584
I20260808 04:51:02.747068 28063 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:02 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:38762
I20260808 04:51:02.747170 28461 tablet_replica.cc:330] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: stopping tablet replica
I20260808 04:51:02.747212 28462 tablet_replica.cc:330] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: stopping tablet replica
I20260808 04:51:02.747221 28460 tablet_replica.cc:330] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: stopping tablet replica
I20260808 04:51:02.747285 28461 raft_consensus.cc:2229] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:51:02.747308 28462 raft_consensus.cc:2229] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:02.747318 28460 raft_consensus.cc:2229] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:02.747418 28462 raft_consensus.cc:2258] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:02.747431 28460 raft_consensus.cc:2258] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:02.747558 28461 raft_consensus.cc:2258] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:02.748123 28460 ts_tablet_manager.cc:1905] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:02.748307 28461 ts_tablet_manager.cc:1905] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:02.748648 28462 ts_tablet_manager.cc:1905] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:02.750751 28460 ts_tablet_manager.cc:1918] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:02.750823 28460 log.cc:1196] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/3f1e0f3df4bb4a09b2e2bfd328e236ac
I20260808 04:51:02.750928 28461 ts_tablet_manager.cc:1918] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:02.751011 28461 log.cc:1196] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/3f1e0f3df4bb4a09b2e2bfd328e236ac
I20260808 04:51:02.751102 28460 ts_tablet_manager.cc:1939] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P a3f9a8a059d84f6a9b8b40c559632c12: Deleting consensus metadata
I20260808 04:51:02.751132 28462 ts_tablet_manager.cc:1918] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:02.751204 28462 log.cc:1196] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/3f1e0f3df4bb4a09b2e2bfd328e236ac
I20260808 04:51:02.751291 28461 ts_tablet_manager.cc:1939] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P c8c213451a804d5aa332b236ff079df7: Deleting consensus metadata
I20260808 04:51:02.751466 28462 ts_tablet_manager.cc:1939] T 3f1e0f3df4bb4a09b2e2bfd328e236ac P 16770a1d4516461bb4a6ec62ace82a18: Deleting consensus metadata
I20260808 04:51:02.751830 27633 catalog_manager.cc:4841] TS a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389): tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac (table test-table [id=f5c902312acb45e29e182ebfe2d0c5e8]) successfully deleted
I20260808 04:51:02.752233 27632 catalog_manager.cc:4841] TS 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685): tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac (table test-table [id=f5c902312acb45e29e182ebfe2d0c5e8]) successfully deleted
I20260808 04:51:02.752493 27649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:51:02.752794 27631 catalog_manager.cc:4841] TS c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829): tablet 3f1e0f3df4bb4a09b2e2bfd328e236ac (table test-table [id=f5c902312acb45e29e182ebfe2d0c5e8]) successfully deleted
W20260808 04:51:02.752887 27649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:51:02.757277 27860 tablet_service.cc:1467] Processing CreateTablet for tablet aaeb0290d0b9488e9d78cbaae14e2efe (DEFAULT_TABLE table=test-table [id=17fbebe65dd24c348ef0b182269ee696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:02.757432 28063 tablet_service.cc:1467] Processing CreateTablet for tablet aaeb0290d0b9488e9d78cbaae14e2efe (DEFAULT_TABLE table=test-table [id=17fbebe65dd24c348ef0b182269ee696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:02.757531 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeb0290d0b9488e9d78cbaae14e2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.757568 28279 tablet_service.cc:1467] Processing CreateTablet for tablet aaeb0290d0b9488e9d78cbaae14e2efe (DEFAULT_TABLE table=test-table [id=17fbebe65dd24c348ef0b182269ee696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:02.757686 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeb0290d0b9488e9d78cbaae14e2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.757800 28279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeb0290d0b9488e9d78cbaae14e2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:02.759832 28420 tablet_bootstrap.cc:492] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7: Bootstrap starting.
I20260808 04:51:02.759914 28421 tablet_bootstrap.cc:492] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: Bootstrap starting.
I20260808 04:51:02.760690 28420 tablet_bootstrap.cc:654] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.760741 28421 tablet_bootstrap.cc:654] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.762014 28421 tablet_bootstrap.cc:492] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: No bootstrap required, opened a new log
I20260808 04:51:02.762084 28421 ts_tablet_manager.cc:1397] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:51:02.762452 28421 raft_consensus.cc:348] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.762563 28421 raft_consensus.cc:374] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.762595 28421 raft_consensus.cc:729] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16770a1d4516461bb4a6ec62ace82a18, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.762717 28421 consensus_queue.cc:260] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.762959 28421 ts_tablet_manager.cc:1428] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:51:02.763571 28423 tablet_bootstrap.cc:492] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: Bootstrap starting.
I20260808 04:51:02.764457 28423 tablet_bootstrap.cc:654] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:02.765554 28420 tablet_bootstrap.cc:492] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7: No bootstrap required, opened a new log
I20260808 04:51:02.765554 28423 tablet_bootstrap.cc:492] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: No bootstrap required, opened a new log
I20260808 04:51:02.765642 28423 ts_tablet_manager.cc:1397] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 04:51:02.765642 28420 ts_tablet_manager.cc:1397] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 04:51:02.765998 28423 raft_consensus.cc:348] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.765998 28420 raft_consensus.cc:348] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.766099 28423 raft_consensus.cc:374] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.766111 28420 raft_consensus.cc:374] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:02.766139 28423 raft_consensus.cc:729] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f9a8a059d84f6a9b8b40c559632c12, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.766146 28420 raft_consensus.cc:729] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c213451a804d5aa332b236ff079df7, State: Initialized, Role: FOLLOWER
I20260808 04:51:02.766268 28423 consensus_queue.cc:260] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.766268 28420 consensus_queue.cc:260] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:02.766525 28420 ts_tablet_manager.cc:1428] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:51:02.766523 28423 ts_tablet_manager.cc:1428] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:51:03.060894 28432 raft_consensus.cc:482] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:51:03.061074 28432 raft_consensus.cc:504] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:03.061578 28432 leader_election.cc:288] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829), a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389)
I20260808 04:51:03.062031 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0290d0b9488e9d78cbaae14e2efe" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c213451a804d5aa332b236ff079df7" is_pre_election: true
I20260808 04:51:03.062028 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0290d0b9488e9d78cbaae14e2efe" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" is_pre_election: true
I20260808 04:51:03.062230 27891 raft_consensus.cc:2454] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 0.
I20260808 04:51:03.062278 28097 raft_consensus.cc:2454] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 0.
I20260808 04:51:03.062649 28204 leader_election.cc:302] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:03.062834 28432 raft_consensus.cc:2781] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:51:03.062901 28432 raft_consensus.cc:482] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:51:03.062929 28432 raft_consensus.cc:3037] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:03.063920 28432 raft_consensus.cc:504] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:03.064263 28432 leader_election.cc:288] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 election: Requested vote from peers c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829), a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389)
I20260808 04:51:03.064623 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0290d0b9488e9d78cbaae14e2efe" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c213451a804d5aa332b236ff079df7"
I20260808 04:51:03.064694 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0290d0b9488e9d78cbaae14e2efe" candidate_uuid: "16770a1d4516461bb4a6ec62ace82a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f9a8a059d84f6a9b8b40c559632c12"
I20260808 04:51:03.064740 27891 raft_consensus.cc:3037] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:03.064836 28097 raft_consensus.cc:3037] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:03.065680 27891 raft_consensus.cc:2454] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 1.
I20260808 04:51:03.065828 28097 raft_consensus.cc:2454] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16770a1d4516461bb4a6ec62ace82a18 in term 1.
I20260808 04:51:03.065968 28204 leader_election.cc:302] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16770a1d4516461bb4a6ec62ace82a18, c8c213451a804d5aa332b236ff079df7; no voters:
I20260808 04:51:03.066155 28432 raft_consensus.cc:2781] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:03.066296 28432 raft_consensus.cc:686] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 1 LEADER]: Becoming Leader. State: Replica: 16770a1d4516461bb4a6ec62ace82a18, State: Running, Role: LEADER
I20260808 04:51:03.066408 28432 consensus_queue.cc:237] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } }
I20260808 04:51:03.068061 27650 catalog_manager.cc:5495] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67). New cstate: current_term: 1 leader_uuid: "16770a1d4516461bb4a6ec62ace82a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16770a1d4516461bb4a6ec62ace82a18" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 45685 } health_report { overall_health: HEALTHY } } }
I20260808 04:51:03.137012 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:03.133728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.15.30.67:45685)
0808 04:51:03.133987 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:03.133994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:03.134008 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:51:03.134048 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:03.134050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:03.134100 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:03.134200 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:03.134465 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:03.134995 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:03.135836 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:03.136131 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:03.136252 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:03.136534 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:03.136617 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:03.136704 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:03.136796 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 04:51:03.136850 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260808 04:51:03.143695 27891 raft_consensus.cc:1261] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Refusing update from remote peer 16770a1d4516461bb4a6ec62ace82a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:03.143702 28097 raft_consensus.cc:1261] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Refusing update from remote peer 16770a1d4516461bb4a6ec62ace82a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:03.144061 28432 consensus_queue.cc:1035] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3f9a8a059d84f6a9b8b40c559632c12" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:03.144512 28432 consensus_queue.cc:1035] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c213451a804d5aa332b236ff079df7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:03.177960 27650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:03.178112 27650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:03.179962 27650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e88ffa9b72dc4653b88477fe180a6c73: Sending DeleteTablet for 3 replicas of tablet aaeb0290d0b9488e9d78cbaae14e2efe
I20260808 04:51:03.180511 27860 tablet_service.cc:1514] Processing DeleteTablet for tablet aaeb0290d0b9488e9d78cbaae14e2efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:03 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39694
I20260808 04:51:03.180601 28279 tablet_service.cc:1514] Processing DeleteTablet for tablet aaeb0290d0b9488e9d78cbaae14e2efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:03 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39584
I20260808 04:51:03.180589 28063 tablet_service.cc:1514] Processing DeleteTablet for tablet aaeb0290d0b9488e9d78cbaae14e2efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:03 UTC) from {username='oryx', principal='oryx/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:38762
I20260808 04:51:03.180708 28461 tablet_replica.cc:330] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7: stopping tablet replica
I20260808 04:51:03.180763 28462 tablet_replica.cc:330] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: stopping tablet replica
I20260808 04:51:03.180776 28460 tablet_replica.cc:330] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: stopping tablet replica
I20260808 04:51:03.180824 28461 raft_consensus.cc:2229] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:03.180871 28462 raft_consensus.cc:2229] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:51:03.180891 28460 raft_consensus.cc:2229] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:03.180927 28461 raft_consensus.cc:2258] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:03.180994 28460 raft_consensus.cc:2258] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:03.181035 28462 raft_consensus.cc:2258] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:03.181704 28462 ts_tablet_manager.cc:1905] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:03.181813 28460 ts_tablet_manager.cc:1905] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:03.181850 28461 ts_tablet_manager.cc:1905] T aaeb0290d0b9488e9d78cbaae14e2efe P c8c213451a804d5aa332b236ff079df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:03.182073 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 27768
I20260808 04:51:03.184927 28460 ts_tablet_manager.cc:1918] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:51:03.185002 28460 log.cc:1196] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/aaeb0290d0b9488e9d78cbaae14e2efe
I20260808 04:51:03.185225 28460 ts_tablet_manager.cc:1939] T aaeb0290d0b9488e9d78cbaae14e2efe P a3f9a8a059d84f6a9b8b40c559632c12: Deleting consensus metadata
I20260808 04:51:03.185889 27633 catalog_manager.cc:4841] TS a3f9a8a059d84f6a9b8b40c559632c12 (127.15.30.66:42389): tablet aaeb0290d0b9488e9d78cbaae14e2efe (table test-table [id=17fbebe65dd24c348ef0b182269ee696]) successfully deleted
I20260808 04:51:03.193424 28462 ts_tablet_manager.cc:1918] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:51:03.193504 28462 log.cc:1196] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/aaeb0290d0b9488e9d78cbaae14e2efe
I20260808 04:51:03.193751 28462 ts_tablet_manager.cc:1939] T aaeb0290d0b9488e9d78cbaae14e2efe P 16770a1d4516461bb4a6ec62ace82a18: Deleting consensus metadata
I20260808 04:51:03.194645 27632 catalog_manager.cc:4841] TS 16770a1d4516461bb4a6ec62ace82a18 (127.15.30.67:45685): tablet aaeb0290d0b9488e9d78cbaae14e2efe (table test-table [id=17fbebe65dd24c348ef0b182269ee696]) successfully deleted
W20260808 04:51:03.201519 27631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:46829 (error 108)
I20260808 04:51:03.202288 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 27973
I20260808 04:51:03.210675 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:03.202177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46829 (local address 127.0.0.1:39726)
0808 04:51:03.210494 (+ 8317us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:03.210580 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":54,"threads_started":1}
W20260808 04:51:03.210834 27631 catalog_manager.cc:4584] TS c8c213451a804d5aa332b236ff079df7 (127.15.30.65:46829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aaeb0290d0b9488e9d78cbaae14e2efe: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46829: connect: Connection refused (error 111)
I20260808 04:51:03.216584 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 28184
I20260808 04:51:03.227995 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 27610
2026-08-08T04:51:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3996 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:51:03 dist-test-slave-zpvx krb5kdc[28578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:51:03 dist-test-slave-zpvx krb5kdc[28578](info): set up 2 sockets
Aug 08 04:51:03 dist-test-slave-zpvx krb5kdc[28578](info): commencing operation
krb5kdc: starting...
W20260808 04:51:04.483327 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:51:04 dist-test-slave-zpvx krb5kdc[28578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:51:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:51:04.612097 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44183
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:35823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:44183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:04.727326 28607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:04.727591 28607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:04.727669 28607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:04.731566 28607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:51:04.731647 28607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:04.731686 28607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:04.731747 28607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:51:04.731787 28607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:51:04.736805 28607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:35823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:44183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:04.737779 28607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:04.738755 28607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:04.747152 28614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:04.747162 28615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:04.747169 28619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:04.747493 28607 server_base.cc:1034] running on GCE node
I20260808 04:51:04.748128 28607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:04.749366 28607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:04.750568 28607 hybrid_clock.cc:648] HybridClock initialized: now 1786164664750552 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T04:51:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1526 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:51:04 dist-test-slave-zpvx krb5kdc[28625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:51:04 dist-test-slave-zpvx krb5kdc[28625](info): set up 2 sockets
Aug 08 04:51:04 dist-test-slave-zpvx krb5kdc[28625](info): commencing operation
krb5kdc: starting...
W20260808 04:51:06.062282 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:51:06 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:51:06.190743 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:40593
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:06.300076 28650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:06.300329 28650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:06.300385 28650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:06.304220 28650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:51:06.304267 28650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:06.304286 28650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:06.304302 28650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:51:06.304317 28650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:51:06.309314 28650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:06.310178 28650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:06.311090 28650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:06.319552 28660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:06.319623 28650 server_base.cc:1034] running on GCE node
W20260808 04:51:06.319583 28665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:06.319568 28659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:06.320437 28650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:06.321494 28650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:06.322681 28650 hybrid_clock.cc:648] HybridClock initialized: now 1786164666322655 us; error 42 us; skew 500 ppm
Aug 08 04:51:06 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164666, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:06.329623 28650 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:51:06.330499 28650 webserver.cc:460] Webserver started at http://127.15.30.126:34867/ using document root <none> and password file <none>
I20260808 04:51:06.331040 28650 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:06.331130 28650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:06.331362 28650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:06.333441 28650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "d027bed569ce4b4dfd7448413c415b3f"
server_key_iv: "5db8e43d27a984802ee1f7ea60fa3ca4"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.334005 28650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "d027bed569ce4b4dfd7448413c415b3f"
server_key_iv: "5db8e43d27a984802ee1f7ea60fa3ca4"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.337642 28650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:51:06.339933 28673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:06.341122 28650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:51:06.341274 28650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "d027bed569ce4b4dfd7448413c415b3f"
server_key_iv: "5db8e43d27a984802ee1f7ea60fa3ca4"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.341380 28650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:06.349858 28650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:06.352350 28650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:06.352599 28650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:06.359521 28650 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:40593
I20260808 04:51:06.359555 28750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:40593 every 8 connection(s)
I20260808 04:51:06.360407 28650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:51:06.362941 28752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:06.366192 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 28650
I20260808 04:51:06.366357 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:51:06.366652 15481 external_mini_cluster.cc:1428] Setting key fa0d94ff43e46167d75e626b166b7115
I20260808 04:51:06.368782 28752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Bootstrap starting.
I20260808 04:51:06.371447 28752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:06.372298 28752 log.cc:824] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:06.374454 28752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: No bootstrap required, opened a new log
Aug 08 04:51:06 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:51:06.378645 28752 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:06.378927 28752 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:06.378978 28752 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cc7287d0be490c9e7a7fc35fa2b33a, State: Initialized, Role: FOLLOWER
I20260808 04:51:06.379422 28752 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:06.379596 28752 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:51:06.379660 28752 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:51:06.379769 28752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:06.380909 28752 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:06.381273 28752 leader_election.cc:302] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5cc7287d0be490c9e7a7fc35fa2b33a; no voters:
I20260808 04:51:06.381587 28752 leader_election.cc:288] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:51:06.381744 28760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:06.381963 28760 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 LEADER]: Becoming Leader. State: Replica: f5cc7287d0be490c9e7a7fc35fa2b33a, State: Running, Role: LEADER
I20260808 04:51:06.382288 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:06.368025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.15.30.126:40593)
0808 04:51:06.368461 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:06.368482 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:06.368523 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:51:06.369380 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:06.369403 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:06.369807 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:06.370178 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:06.370862 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.372334 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.373442 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.374107 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.374264 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:06.376852 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:06.376859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:06.376880 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:06.376919 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:06.378961 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:06.379470 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:06.379476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:06.379481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:06.379628 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:06.380035 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:06.380039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:06.380041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:06.380557 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:06.380790 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:06.381009 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 04:51:06.381299 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260808 04:51:06.382303 28760 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:06.382829 28752 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:51:06.383955 28761 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } } }
I20260808 04:51:06.384073 28761 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:06.384423 28763 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5cc7287d0be490c9e7a7fc35fa2b33a. Latest consensus state: current_term: 1 leader_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } } }
I20260808 04:51:06.384511 28763 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:06.385432 28775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:51:06.387830 28775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:51:06.393532 28775 catalog_manager.cc:1347] Generated new cluster ID: 08518d9cd4af43618624d21dc1e95e33
I20260808 04:51:06.393592 28775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:51:06.427757 28775 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:51:06.429148 28775 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:51:06.442767 28775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Generated new TSK 0
I20260808 04:51:06.443503 28775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:51:06.483965 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40593
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:06.612398 28790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:06.612624 28790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:06.612664 28790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:06.616577 28790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:06.616623 28790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:06.616677 28790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:51:06.621567 28790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:06.622457 28790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:06.623373 28790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:06.632558 28800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:06.632584 28797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:06.632773 28790 server_base.cc:1034] running on GCE node
W20260808 04:51:06.632869 28804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:06.633338 28790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:06.633953 28790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:06.635118 28790 hybrid_clock.cc:648] HybridClock initialized: now 1786164666635088 us; error 42 us; skew 500 ppm
Aug 08 04:51:06 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164666, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:06.637907 28790 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:51:06.638664 28790 webserver.cc:460] Webserver started at http://127.15.30.65:42567/ using document root <none> and password file <none>
I20260808 04:51:06.639155 28790 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:06.639222 28790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:06.639380 28790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:06.641695 28790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "297478a1af04481db9ec15a4b6554114"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "acbe7b9a8e460f90a1139ce84f14c8bc"
server_key_iv: "fec1900c7f5df02db3c25eb6eac97e36"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.642194 28790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "297478a1af04481db9ec15a4b6554114"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "acbe7b9a8e460f90a1139ce84f14c8bc"
server_key_iv: "fec1900c7f5df02db3c25eb6eac97e36"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.645892 28790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:51:06.648278 28819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:06.649474 28790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:51:06.649605 28790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "297478a1af04481db9ec15a4b6554114"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "acbe7b9a8e460f90a1139ce84f14c8bc"
server_key_iv: "fec1900c7f5df02db3c25eb6eac97e36"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.649709 28790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:06.660519 28790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:06.663101 28790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:06.663295 28790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:06.663851 28790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:06.664687 28790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:06.664763 28790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:06.664849 28790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:06.664896 28790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:06.673686 28790 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:38975
I20260808 04:51:06.673720 29005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:38975 every 8 connection(s)
I20260808 04:51:06.674561 28790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:51:06.680964 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 28790
I20260808 04:51:06.681121 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:51:06.681435 15481 external_mini_cluster.cc:1428] Setting key 869451b0a46c25ba8b39b6c2653ee296
Aug 08 04:51:06 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164666, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:51:06.690565 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:06.676639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:53773)
0808 04:51:06.677088 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:06.677128 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:06.678296 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:06.678864 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:06.678871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:06.679226 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:06.679660 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:06.679671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.680527 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.680535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:06.681517 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:06.681531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.682040 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.682044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:06.682214 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:06.682850 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:06.682867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:06.684572 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:06.687130 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:06.687137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:06.687153 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:06.687532 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:06.687904 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:06.687910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:06.687912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:06.688103 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:06.688626 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:06.688635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:06.688965 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 04:51:06.689189 (+ 224us) client_negotiation.cc:240] Negotiation successful
0808 04:51:06.689525 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20260808 04:51:06.691672 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:06.676309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:53773 (local address 127.15.30.126:40593)
0808 04:51:06.676463 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:06.676467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:06.677242 (+ 775us) server_negotiation.cc:407] Connection header received
0808 04:51:06.678501 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:06.678505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:06.678565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:06.678673 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:06.679806 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.680343 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.681652 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.681889 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.682002 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:06.684792 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:06.684799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:06.684802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:06.684831 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:06.686982 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:06.687663 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:06.687666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:06.687668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:06.687781 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:06.688213 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:06.688216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:06.688218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:06.688507 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:06.688625 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:06.691302 (+ 2677us) server_negotiation.cc:299] Negotiation successful
0808 04:51:06.691451 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:51:06.692956 29006 heartbeater.cc:347] Connected to a master server at 127.15.30.126:40593
I20260808 04:51:06.693300 29006 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:06.693874 29006 heartbeater.cc:511] Master 127.15.30.126:40593 requested a full tablet report, sending...
I20260808 04:51:06.695506 28697 ts_manager.cc:194] Registered new tserver with Master: 297478a1af04481db9ec15a4b6554114 (127.15.30.65:38975)
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:51:06.696769 28697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:53773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:51:06.714684 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40593
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:06.824327 29016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:06.824572 29016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:06.824612 29016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:06.828503 29016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:06.828555 29016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:06.828609 29016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:51:06.833648 29016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:06.834615 29016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:06.835597 29016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:06.844997 29029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:06.844939 29025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:06.844972 29016 server_base.cc:1034] running on GCE node
W20260808 04:51:06.845002 29026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:06.845570 29016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:06.846237 29016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:06.847393 29016 hybrid_clock.cc:648] HybridClock initialized: now 1786164666847360 us; error 47 us; skew 500 ppm
Aug 08 04:51:06 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164666, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:06.850168 29016 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:51:06.851038 29016 webserver.cc:460] Webserver started at http://127.15.30.66:34375/ using document root <none> and password file <none>
I20260808 04:51:06.851536 29016 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:06.851593 29016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:06.851764 29016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:06.853765 29016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "321011871905484bbf1286f199b16e45"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "1f36cd4bfd66ec9bdad68800eddaf3db"
server_key_iv: "2f08f8536688d4c3339974ee870c6f07"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.854265 29016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "321011871905484bbf1286f199b16e45"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "1f36cd4bfd66ec9bdad68800eddaf3db"
server_key_iv: "2f08f8536688d4c3339974ee870c6f07"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.857792 29016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:51:06.860049 29037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:06.861260 29016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:51:06.861392 29016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "321011871905484bbf1286f199b16e45"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "1f36cd4bfd66ec9bdad68800eddaf3db"
server_key_iv: "2f08f8536688d4c3339974ee870c6f07"
server_key_version: "encryptionkey@0"
I20260808 04:51:06.861498 29016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:06.874205 29016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:06.877153 29016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:06.877347 29016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:06.877912 29016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:06.878715 29016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:06.878784 29016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:06.878859 29016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:06.878909 29016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:06.888828 29016 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:39445
I20260808 04:51:06.888878 29206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:39445 every 8 connection(s)
I20260808 04:51:06.889717 29016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:51:06.890612 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 29016
I20260808 04:51:06.890753 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:51:06.891038 15481 external_mini_cluster.cc:1428] Setting key 351ce761d74cc6b1f0fca22ac7f0d9f1
Aug 08 04:51:06 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164666, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:51:06.904716 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:06.891601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:46807 (local address 127.15.30.126:40593)
0808 04:51:06.891731 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:06.891736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:06.892970 (+ 1234us) server_negotiation.cc:407] Connection header received
0808 04:51:06.893879 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:06.893882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:06.893938 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:06.894017 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:06.895503 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.896042 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.897886 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.898122 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.898210 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:06.901086 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:06.901090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:06.901092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:06.901117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:06.902626 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:06.903153 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:06.903156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:06.903158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:06.903238 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:06.903588 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:06.903592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:06.903594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:06.903785 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:06.903898 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:06.904417 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 04:51:06.904543 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
I20260808 04:51:06.905465 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:06.892068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:46807)
0808 04:51:06.892828 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:06.892878 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:06.893674 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:06.894179 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:06.894187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:06.894657 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:06.895333 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:06.895355 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.896175 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.896189 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:06.897737 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:06.897752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:06.898293 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:06.898297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:06.898531 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:06.899126 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:06.899151 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:06.900902 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:06.902748 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:06.902754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:06.902768 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:06.903027 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:06.903335 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:06.903338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:06.903340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:06.903480 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:06.903897 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:06.903905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:06.904144 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 04:51:06.904325 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:51:06.904666 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":139,"threads_started":1}
I20260808 04:51:06.906656 29208 heartbeater.cc:347] Connected to a master server at 127.15.30.126:40593
I20260808 04:51:06.906903 29208 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:06.907382 29208 heartbeater.cc:511] Master 127.15.30.126:40593 requested a full tablet report, sending...
I20260808 04:51:06.908481 28697 ts_manager.cc:194] Registered new tserver with Master: 321011871905484bbf1286f199b16e45 (127.15.30.66:39445)
I20260808 04:51:06.909397 28697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:46807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:51:06.920979 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40593
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:07.032645 29216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:07.032871 29216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:07.032910 29216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:07.036798 29216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:07.036842 29216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:07.036896 29216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:51:07.041883 29216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:07.042757 29216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:07.043658 29216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:07.052961 29230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:07.053033 29224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.053149 29216 server_base.cc:1034] running on GCE node
W20260808 04:51:07.053028 29227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.053660 29216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:07.054441 29216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:07.055644 29216 hybrid_clock.cc:648] HybridClock initialized: now 1786164667055629 us; error 24 us; skew 500 ppm
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164667, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:07.058745 29216 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:51:07.059589 29216 webserver.cc:460] Webserver started at http://127.15.30.67:36593/ using document root <none> and password file <none>
I20260808 04:51:07.060134 29216 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:07.060196 29216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:07.060357 29216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:07.062462 29216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "51482ae5154048ceb2573f074249f3d5"
format_stamp: "Formatted at 2026-08-08 04:51:07 on dist-test-slave-zpvx"
server_key: "9cf47f7e5a654d2e0c1839afbaf07814"
server_key_iv: "d1de5ec2000984ac71b17b47f3334185"
server_key_version: "encryptionkey@0"
I20260808 04:51:07.062989 29216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "51482ae5154048ceb2573f074249f3d5"
format_stamp: "Formatted at 2026-08-08 04:51:07 on dist-test-slave-zpvx"
server_key: "9cf47f7e5a654d2e0c1839afbaf07814"
server_key_iv: "d1de5ec2000984ac71b17b47f3334185"
server_key_version: "encryptionkey@0"
I20260808 04:51:07.067134 29216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 04:51:07.069615 29241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.070789 29216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:51:07.070927 29216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "51482ae5154048ceb2573f074249f3d5"
format_stamp: "Formatted at 2026-08-08 04:51:07 on dist-test-slave-zpvx"
server_key: "9cf47f7e5a654d2e0c1839afbaf07814"
server_key_iv: "d1de5ec2000984ac71b17b47f3334185"
server_key_version: "encryptionkey@0"
I20260808 04:51:07.071031 29216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:07.088688 29216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:07.091900 29216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:07.092110 29216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:07.092803 29216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:07.093626 29216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:07.093719 29216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.093802 29216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:07.093842 29216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.103173 29216 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:36657
I20260808 04:51:07.103196 29410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:36657 every 8 connection(s)
I20260808 04:51:07.104058 29216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:51:07.107095 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 29216
I20260808 04:51:07.107242 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:51:07.107589 15481 external_mini_cluster.cc:1428] Setting key b6de5554704f67042632138590da523e
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164667, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:51:07.120818 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.105893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:48483 (local address 127.15.30.126:40593)
0808 04:51:07.106066 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.106071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.107170 (+ 1099us) server_negotiation.cc:407] Connection header received
0808 04:51:07.108543 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.108546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.108611 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.108710 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.110497 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.111428 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.113127 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.113492 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.113617 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.116853 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:07.116857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:07.116860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:07.116889 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:07.118836 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:07.119440 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:07.119443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:07.119444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:07.119523 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:07.119794 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:07.119796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:07.119797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:07.119957 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:07.120030 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:07.120571 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 04:51:07.120667 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:51:07.121723 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.106289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:48483)
0808 04:51:07.107001 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.107062 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.108220 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.108926 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.108937 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.109524 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.110307 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.110326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.111572 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.111579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.112961 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.112973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.113695 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.113703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.113948 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.114636 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.114662 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.116668 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:07.118985 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:07.118993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:07.119009 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:07.119330 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:07.119587 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:07.119589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:07.119590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:07.119711 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:07.120045 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:07.120050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:07.120301 (+ 251us) client_negotiation.cc:769] Sending connection context
0808 04:51:07.120517 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 04:51:07.120807 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":110,"threads_started":1}
I20260808 04:51:07.122793 29411 heartbeater.cc:347] Connected to a master server at 127.15.30.126:40593
I20260808 04:51:07.123068 29411 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:07.123592 29411 heartbeater.cc:511] Master 127.15.30.126:40593 requested a full tablet report, sending...
I20260808 04:51:07.124619 28697 ts_manager.cc:194] Registered new tserver with Master: 51482ae5154048ceb2573f074249f3d5 (127.15.30.67:36657)
I20260808 04:51:07.125118 28697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:48483
I20260808 04:51:07.134853 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab.
Entry for principal oryx/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab.
WARNING: no policy specified for oryx/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab.
Entry for principal oryx/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab.
WARNING: no policy specified for oryx/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab.
Entry for principal oryx/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab.
I20260808 04:51:07.206930 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 28790
I20260808 04:51:07.220877 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 29016
I20260808 04:51:07.232761 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 29216
I20260808 04:51:07.239071 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 28650
I20260808 04:51:07.248750 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:40593
--webserver_interface=127.15.30.126
--webserver_port=34867
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:07.357592 29427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:07.357823 29427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:07.357896 29427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:07.361804 29427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:51:07.361850 29427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:07.361868 29427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:07.361929 29427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:51:07.361965 29427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:51:07.366961 29427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=34867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:07.367856 29427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:07.368810 29427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:07.376721 29438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:07.376747 29434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.376855 29427 server_base.cc:1034] running on GCE node
W20260808 04:51:07.376756 29432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.377509 29427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:07.378515 29427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:07.379703 29427 hybrid_clock.cc:648] HybridClock initialized: now 1786164667379692 us; error 28 us; skew 500 ppm
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164667, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:07.382436 29427 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:51:07.383286 29427 webserver.cc:460] Webserver started at http://127.15.30.126:34867/ using document root <none> and password file <none>
I20260808 04:51:07.383826 29427 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:07.383936 29427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:07.387485 29427 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:07.389525 29446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.390630 29427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:51:07.390767 29427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "d027bed569ce4b4dfd7448413c415b3f"
server_key_iv: "5db8e43d27a984802ee1f7ea60fa3ca4"
server_key_version: "encryptionkey@0"
I20260808 04:51:07.391266 29427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:07.410418 29427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:07.413777 29427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:07.413985 29427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:07.421176 29427 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:40593
I20260808 04:51:07.421208 29528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:40593 every 8 connection(s)
I20260808 04:51:07.422199 29427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:51:07.424744 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 29427
I20260808 04:51:07.425216 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:38975
--local_ip_for_outbound_sockets=127.15.30.65
--tserver_master_addrs=127.15.30.126:40593
--webserver_port=42567
--webserver_interface=127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:51:07.426563 29529 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:51:07.428961 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Bootstrap starting.
I20260808 04:51:07.445452 29529 log.cc:824] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:07.451850 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:51:07.452220 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Bootstrap complete.
I20260808 04:51:07.456512 29529 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:07.456961 29529 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cc7287d0be490c9e7a7fc35fa2b33a, State: Initialized, Role: FOLLOWER
I20260808 04:51:07.457439 29529 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:07.457588 29529 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:51:07.457679 29529 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:51:07.457798 29529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:51:07.459647 29529 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:07.459982 29529 leader_election.cc:302] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5cc7287d0be490c9e7a7fc35fa2b33a; no voters:
I20260808 04:51:07.460350 29529 leader_election.cc:288] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:51:07.460494 29535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:51:07.460717 29535 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [term 2 LEADER]: Becoming Leader. State: Replica: f5cc7287d0be490c9e7a7fc35fa2b33a, State: Running, Role: LEADER
I20260808 04:51:07.461005 29535 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } }
I20260808 04:51:07.461575 29529 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:51:07.461941 29535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5cc7287d0be490c9e7a7fc35fa2b33a. Latest consensus state: current_term: 2 leader_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } } }
I20260808 04:51:07.462052 29535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:07.462317 29536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cc7287d0be490c9e7a7fc35fa2b33a" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40593 } } }
I20260808 04:51:07.462415 29536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:07.462509 29539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:51:07.465919 29539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:51:07.466707 29539 catalog_manager.cc:1259] Loaded cluster ID: 08518d9cd4af43618624d21dc1e95e33
I20260808 04:51:07.466764 29539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:51:07.469132 29539 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:51:07.469791 29539 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f5cc7287d0be490c9e7a7fc35fa2b33a: Loaded TSK: 0
I20260808 04:51:07.470971 29539 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:51:07.540584 29531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:07.540844 29531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:07.540916 29531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:07.544983 29531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:07.545060 29531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:07.545154 29531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:51:07.550246 29531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.65.keytab
--principal=oryx/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:38975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=42567
--tserver_master_addrs=127.15.30.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:07.551190 29531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:07.552208 29531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:07.562597 29572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:07.562580 29571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.563014 29531 server_base.cc:1034] running on GCE node
W20260808 04:51:07.562707 29578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.563566 29531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:07.564218 29531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:07.565414 29531 hybrid_clock.cc:648] HybridClock initialized: now 1786164667565406 us; error 48 us; skew 500 ppm
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164667, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:07.568269 29531 init.cc:377] Logged in from keytab as oryx/127.15.30.65@KRBTEST.COM (short username oryx)
I20260808 04:51:07.569129 29531 webserver.cc:460] Webserver started at http://127.15.30.65:42567/ using document root <none> and password file <none>
I20260808 04:51:07.569679 29531 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:07.569782 29531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:07.573773 29531 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:51:07.575860 29593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.577139 29531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:07.577270 29531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "297478a1af04481db9ec15a4b6554114"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "acbe7b9a8e460f90a1139ce84f14c8bc"
server_key_iv: "fec1900c7f5df02db3c25eb6eac97e36"
server_key_version: "encryptionkey@0"
I20260808 04:51:07.577795 29531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:07.591161 29531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:07.594452 29531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:07.594650 29531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:07.595305 29531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:07.596143 29531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:07.596211 29531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.596280 29531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:07.596333 29531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.606220 29531 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:38975
I20260808 04:51:07.606279 29760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:38975 every 8 connection(s)
I20260808 04:51:07.607203 29531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:51:07.612712 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 29531
I20260808 04:51:07.613267 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:39445
--local_ip_for_outbound_sockets=127.15.30.66
--tserver_master_addrs=127.15.30.126:40593
--webserver_port=34375
--webserver_interface=127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:07.623694 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.609652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:47045)
0808 04:51:07.611875 (+ 2223us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.611915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.612901 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.614117 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.614129 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.614647 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.615321 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.615338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.617254 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.617257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.618124 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.618134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.619328 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.619332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.619489 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.620179 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.620197 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.622862 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2049,"thread_start_us":139,"threads_started":1}
W20260808 04:51:07.624125 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.609564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:47045 (local address 127.15.30.126:40593)
0808 04:51:07.610023 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.610042 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.612031 (+ 1989us) server_negotiation.cc:407] Connection header received
0808 04:51:07.613304 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.613314 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.613685 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.613945 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.615500 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.617069 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.618284 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.619132 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.619329 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.623154 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:47045: BlockingRecv error: recv got EOF from 127.15.30.65:47045 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":114,"threads_started":1}
W20260808 04:51:07.624953 29761 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:40593 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.30.126:40593: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:07.630939 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.625421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:37531)
0808 04:51:07.625627 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.625641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.625718 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.625970 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.625972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.626110 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.626241 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.626248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.627243 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.627246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.627853 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.627861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.628440 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.628448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.628539 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.629020 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.629044 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.630846 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260808 04:51:07.631096 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.625541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:37531 (local address 127.15.30.126:40593)
0808 04:51:07.625694 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.625701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.625719 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:51:07.625791 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.625795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.625850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.625947 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.626350 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.627127 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.627960 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.628296 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.628427 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.630992 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:37531: BlockingRecv error: recv got EOF from 127.15.30.65:37531 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:07.636848 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.631639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:42425)
0808 04:51:07.631875 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.631887 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.631960 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.632185 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.632187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.632313 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.632443 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.632450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.633411 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.633414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.633953 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.633960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.634501 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.634504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.634564 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.635013 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.635027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.636766 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178}
W20260808 04:51:07.637024 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.631759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:42425 (local address 127.15.30.126:40593)
0808 04:51:07.631876 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.631880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.631942 (+ 62us) server_negotiation.cc:407] Connection header received
0808 04:51:07.632031 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.632034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.632076 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.632156 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.632543 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.633301 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.634061 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.634393 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.634491 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.636911 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:42425: BlockingRecv error: recv got EOF from 127.15.30.65:42425 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 04:51:07.637240 29761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:51:07.731058 29765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:07.731307 29765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:07.731349 29765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:07.735360 29765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:07.735419 29765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:07.735476 29765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:51:07.740700 29765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.66.keytab
--principal=oryx/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:39445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=34375
--tserver_master_addrs=127.15.30.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:07.741786 29765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:07.742825 29765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:07.752874 29772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:07.752880 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:07.752882 29770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.753232 29765 server_base.cc:1034] running on GCE node
I20260808 04:51:07.753638 29765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:07.754284 29765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:07.755478 29765 hybrid_clock.cc:648] HybridClock initialized: now 1786164667755444 us; error 56 us; skew 500 ppm
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164667, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:07.759209 29765 init.cc:377] Logged in from keytab as oryx/127.15.30.66@KRBTEST.COM (short username oryx)
I20260808 04:51:07.760149 29765 webserver.cc:460] Webserver started at http://127.15.30.66:34375/ using document root <none> and password file <none>
I20260808 04:51:07.760799 29765 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:07.760901 29765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:07.765173 29765 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:51:07.767318 29784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.768635 29765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:51:07.768770 29765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "321011871905484bbf1286f199b16e45"
format_stamp: "Formatted at 2026-08-08 04:51:06 on dist-test-slave-zpvx"
server_key: "1f36cd4bfd66ec9bdad68800eddaf3db"
server_key_iv: "2f08f8536688d4c3339974ee870c6f07"
server_key_version: "encryptionkey@0"
I20260808 04:51:07.769340 29765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:07.782944 29765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:07.786181 29765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:07.786379 29765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:07.786962 29765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:07.787777 29765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:07.787847 29765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.787918 29765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:07.787973 29765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.797467 29765 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:39445
I20260808 04:51:07.797497 29946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:39445 every 8 connection(s)
I20260808 04:51:07.798434 29765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:51:07.801478 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 29765
I20260808 04:51:07.802066 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:36657
--local_ip_for_outbound_sockets=127.15.30.67
--tserver_master_addrs=127.15.30.126:40593
--webserver_port=36593
--webserver_interface=127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:51:07.811802 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.800366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:56567 (local address 127.15.30.126:40593)
0808 04:51:07.800554 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.800559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.801864 (+ 1305us) server_negotiation.cc:407] Connection header received
0808 04:51:07.803352 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.803355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.803405 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.803481 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.805015 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.805764 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.807022 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.807314 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.807432 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.811697 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:56567: BlockingRecv error: recv got EOF from 127.15.30.66:56567 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:51:07.812660 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.800831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:56567)
0808 04:51:07.801686 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.801747 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.803114 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.803720 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.803733 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.804280 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.804859 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.804873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.805892 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.805895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.806869 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.806879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.807515 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.807523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.807773 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.808480 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.808506 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.811556 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":108,"threads_started":1}
W20260808 04:51:07.813244 29948 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:40593 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.30.126:40593: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:07.818950 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.813724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:36863)
0808 04:51:07.813819 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.813834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.813931 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.814186 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.814189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.814360 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.814451 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.814456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.815182 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.815187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.815721 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.815729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.816181 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.816185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.816280 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.816801 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.816814 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.818798 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 04:51:07.819325 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.813822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:36863 (local address 127.15.30.126:40593)
0808 04:51:07.813961 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.813968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.813984 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:51:07.814028 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.814031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.814081 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.814170 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.814585 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.815072 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.815849 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.816066 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.816169 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.818923 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:36863: BlockingRecv error: recv got EOF from 127.15.30.66:36863 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:07.824837 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.819481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:34321)
0808 04:51:07.819611 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.819630 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.819709 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.820032 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.820035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.820172 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.820245 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.820250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.821273 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.821276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.821831 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.821839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.822395 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.822398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.822456 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.822933 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.822946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.824749 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 04:51:07.825009 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.819704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:34321 (local address 127.15.30.126:40593)
0808 04:51:07.819826 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.819830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.819843 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:51:07.819879 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.819882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.819919 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.820001 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.820354 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.821144 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.821945 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.822282 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.822382 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.824900 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:34321: BlockingRecv error: recv got EOF from 127.15.30.66:34321 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 04:51:07.825048 29948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:51:07.919997 29952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:07.920239 29952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:07.920280 29952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:07.924098 29952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:07.924144 29952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:07.924199 29952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:51:07.929101 29952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/krb5kdc/oryx_127.15.30.67.keytab
--principal=oryx/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:36657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=36593
--tserver_master_addrs=127.15.30.126:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:07.929980 29952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:07.930888 29952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:07.940035 29962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:07.940117 29971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:07.940188 29963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:07.940344 29952 server_base.cc:1034] running on GCE node
I20260808 04:51:07.940826 29952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:07.941457 29952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:07.942662 29952 hybrid_clock.cc:648] HybridClock initialized: now 1786164667942638 us; error 51 us; skew 500 ppm
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164667, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:07.945464 29952 init.cc:377] Logged in from keytab as oryx/127.15.30.67@KRBTEST.COM (short username oryx)
I20260808 04:51:07.946365 29952 webserver.cc:460] Webserver started at http://127.15.30.67:36593/ using document root <none> and password file <none>
I20260808 04:51:07.946930 29952 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:07.947041 29952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:07.950803 29952 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 04:51:07.952834 29984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.954007 29952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:07.954269 29952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "51482ae5154048ceb2573f074249f3d5"
format_stamp: "Formatted at 2026-08-08 04:51:07 on dist-test-slave-zpvx"
server_key: "9cf47f7e5a654d2e0c1839afbaf07814"
server_key_iv: "d1de5ec2000984ac71b17b47f3334185"
server_key_version: "encryptionkey@0"
I20260808 04:51:07.954909 29952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:07.972259 29952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:07.975327 29952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:07.975513 29952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:07.976071 29952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:07.976930 29952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:07.977002 29952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.977074 29952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:07.977128 29952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:07.986239 29952 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:36657
I20260808 04:51:07.986275 30157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:36657 every 8 connection(s)
I20260808 04:51:07.987191 29952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:51:07.989970 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 29952
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:51:07.998250 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.988919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:46383 (local address 127.15.30.126:40593)
0808 04:51:07.989062 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.989066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.990167 (+ 1101us) server_negotiation.cc:407] Connection header received
0808 04:51:07.991286 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.991290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.991349 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.991450 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.992598 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.993099 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.994335 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.994551 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.994668 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.998153 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:46383: BlockingRecv error: recv got EOF from 127.15.30.67:46383 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:51:07.998919 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.989260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:46383)
0808 04:51:07.990002 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:07.990062 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:07.991078 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:07.991627 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:07.991634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:07.991967 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:07.992465 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.992481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.993232 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.993235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.994188 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:07.994200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.994740 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.994747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:07.994958 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.995489 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:07.995504 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:07.998040 (+ 2536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":116,"threads_started":1}
W20260808 04:51:07.999393 30159 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:40593 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.30.126:40593: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
I20260808 04:51:08.000075 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.991036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.15.30.126:40593)
0808 04:51:07.991272 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:07.991275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:07.991287 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:51:07.991320 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:07.991322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:07.991357 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:07.991459 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:07.992210 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.992811 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.993693 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:07.993915 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:07.994043 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:07.995663 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:07.995668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:07.995681 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:07.995707 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:07.997629 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:07.998086 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:07.998092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:07.998098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:07.998252 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:07.998688 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:07.998691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:07.998693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:07.999136 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:07.999327 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:07.999633 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:51:07.999901 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":167,"spinlock_wait_cycles":24832,"thread_start_us":99,"threads_started":1}
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:07 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:08.004812 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:07.999893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:44337)
0808 04:51:07.999984 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:08.000141 (+ 157us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:08.000266 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:08.000540 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:08.000543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:08.000666 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:08.000746 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.000751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.001488 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.001491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.002187 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.002197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.002657 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.002661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.002754 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.003120 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:08.003131 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:08.004731 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 04:51:08.004963 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:07.999966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:44337 (local address 127.15.30.126:40593)
0808 04:51:08.000074 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:08.000077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:08.000202 (+ 125us) server_negotiation.cc:407] Connection header received
0808 04:51:08.000362 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:08.000389 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:08.000434 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:08.000515 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:08.000852 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.001364 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.002309 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.002530 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.002646 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.004851 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:44337: BlockingRecv error: recv got EOF from 127.15.30.67:44337 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 08 04:51:08 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:08 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:08.009732 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:08.005366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:44541)
0808 04:51:08.005467 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:08.005482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:08.005567 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:08.005772 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:08.005775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:08.005918 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:08.005979 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.005983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.006670 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.006675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.007160 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.007167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.007590 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.007594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.007676 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.008017 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:08.008028 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:08.009606 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:51:08.009814 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:08.005447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:44541 (local address 127.15.30.126:40593)
0808 04:51:08.005578 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:08.005582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:08.005597 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:51:08.005635 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:08.005640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:08.005679 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:08.005757 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:08.006125 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.006567 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.007275 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.007477 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.007580 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.009708 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:44541: BlockingRecv error: recv got EOF from 127.15.30.67:44541 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 04:51:08.009956 30159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 04:51:08 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:08 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:08.643886 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:08.637877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:34985)
0808 04:51:08.638216 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:08.638232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:08.638343 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:08.638617 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:08.638619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:08.638823 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:08.638939 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.638945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.639738 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.639741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.640323 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.640332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.640868 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.640870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.640941 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.641655 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:08.641673 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:08.643765 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":104,"threads_started":1}
W20260808 04:51:08.644083 30168 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:08.637985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:34985 (local address 127.15.30.126:40593)
0808 04:51:08.638231 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:08.638234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:08.638314 (+ 80us) server_negotiation.cc:407] Connection header received
0808 04:51:08.638427 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:08.638429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:08.638475 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:08.638562 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:08.639061 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.639611 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.640511 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.640763 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.640847 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.643904 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:34985: BlockingRecv error: recv got EOF from 127.15.30.65:34985 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
Aug 08 04:51:08 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:08 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:08.831514 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:08.825641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:51113)
0808 04:51:08.825946 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:08.825968 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:08.826156 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:08.826447 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:08.826450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:08.826660 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:08.826753 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.826766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.827536 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.827540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.828118 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:08.828127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.828690 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.828697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:08.828793 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.829286 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:08.829296 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:08.831333 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
W20260808 04:51:08.831578 30168 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:08.825707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:51113 (local address 127.15.30.126:40593)
0808 04:51:08.825912 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:08.825917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:08.826092 (+ 175us) server_negotiation.cc:407] Connection header received
0808 04:51:08.826275 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:08.826278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:08.826333 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:08.826419 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:08.826888 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.827423 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.828244 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:08.828559 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:08.828672 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:08.831461 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:51113: BlockingRecv error: recv got EOF from 127.15.30.66:51113 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 08 04:51:09 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:09 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:09.016067 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:09.010531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:56979)
0808 04:51:09.010797 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:09.010811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:09.010907 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:09.011193 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:09.011196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:09.011380 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:09.011498 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:09.011503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.012278 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.012281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:09.012937 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:09.012945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.013389 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.013392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:09.013460 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:09.013948 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:09.013959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:09.015956 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
W20260808 04:51:09.016319 30168 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:09.010610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:56979 (local address 127.15.30.126:40593)
0808 04:51:09.010810 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:09.010814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:09.010886 (+ 72us) server_negotiation.cc:407] Connection header received
0808 04:51:09.010992 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:09.010994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:09.011045 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:09.011121 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:09.011608 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.012179 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.013049 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.013288 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.013375 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:09.016192 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:56979: BlockingRecv error: recv got EOF from 127.15.30.67:56979 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 08 04:51:09 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:09 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:09.650188 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:09.644708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:35353)
0808 04:51:09.644994 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:09.645007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:09.645094 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:09.645375 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:09.645378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:09.645562 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:09.645659 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:09.645664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.646390 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.646392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:09.646947 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:09.646954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.647375 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.647378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:09.647440 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:09.648000 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:09.648012 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:09.649994 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
W20260808 04:51:09.650262 30174 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:09.644790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35353 (local address 127.15.30.126:40593)
0808 04:51:09.645036 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:09.645039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:09.645052 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:51:09.645204 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:09.645206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:09.645251 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:09.645334 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:09.645759 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.646295 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.647045 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.647279 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.647356 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:09.650123 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:35353: BlockingRecv error: recv got EOF from 127.15.30.65:35353 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
Aug 08 04:51:09 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:09 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:09.838383 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:09.832346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:33593)
0808 04:51:09.832707 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:09.832720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:09.832824 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:09.833055 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:09.833057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:09.833255 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:09.833353 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:09.833358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.834199 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.834202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:09.834932 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:09.834942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.835573 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.835576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:09.835646 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:09.836161 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:09.836172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:09.838242 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
W20260808 04:51:09.838485 30174 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:09.832446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:33593 (local address 127.15.30.126:40593)
0808 04:51:09.832601 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:09.832606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:09.832794 (+ 188us) server_negotiation.cc:407] Connection header received
0808 04:51:09.832904 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:09.832906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:09.832954 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:09.833015 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:09.833521 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.834048 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.835159 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:09.835461 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:09.835548 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:09.838394 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:33593: BlockingRecv error: recv got EOF from 127.15.30.66:33593 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 04:51:10 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:10 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:10.022836 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:10.016891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:37031)
0808 04:51:10.017192 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:10.017207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:10.017313 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:10.017572 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:10.017575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:10.017785 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:10.017899 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:10.017905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.018656 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.018658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:10.019293 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:10.019331 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.019853 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.019857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:10.019961 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:10.020659 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:10.020688 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:10.022697 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
W20260808 04:51:10.022919 30174 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:10.017070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:37031 (local address 127.15.30.126:40593)
0808 04:51:10.017212 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:10.017215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:10.017280 (+ 65us) server_negotiation.cc:407] Connection header received
0808 04:51:10.017402 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:10.017404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:10.017450 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:10.017520 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:10.018008 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.018555 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.019449 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.019712 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.019820 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:10.022826 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:37031: BlockingRecv error: recv got EOF from 127.15.30.67:37031 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 04:51:10 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:10 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:10.656936 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:10.651072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:46377)
0808 04:51:10.651407 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:10.651425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:10.651538 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:10.651781 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:10.651784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:10.651980 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:10.652080 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:10.652085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.652953 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.652956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:10.653533 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:10.653541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.653956 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.653958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:10.654021 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:10.654659 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:10.654693 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:10.656821 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
W20260808 04:51:10.657124 30181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:10.651256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:46377 (local address 127.15.30.126:40593)
0808 04:51:10.651531 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:10.651535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:10.651549 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:51:10.651606 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:10.651608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:10.651650 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:10.651729 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:10.652195 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.652840 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.653642 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.653857 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.653967 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:10.656952 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:46377: BlockingRecv error: recv got EOF from 127.15.30.65:46377 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
Aug 08 04:51:10 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:10 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:10.845260 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:10.839231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:40591)
0808 04:51:10.839712 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:10.839729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:10.839860 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:10.840214 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:10.840217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:10.840553 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:10.840661 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:10.840668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.841450 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.841456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:10.842024 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:10.842034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.842500 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.842504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:10.842601 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:10.843143 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:10.843156 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:10.845139 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":123,"threads_started":1}
W20260808 04:51:10.845409 30181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:10.839419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:40591 (local address 127.15.30.126:40593)
0808 04:51:10.839586 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:10.839592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:10.839838 (+ 246us) server_negotiation.cc:407] Connection header received
0808 04:51:10.839957 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:10.839962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:10.840078 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:10.840180 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:10.840793 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.841335 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.842154 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:10.842380 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:10.842493 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:10.845287 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:40591: BlockingRecv error: recv got EOF from 127.15.30.66:40591 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:51:11 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:11 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:11.029331 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:11.023633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:55475)
0808 04:51:11.023897 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:11.023911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:11.024016 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:11.024263 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:11.024266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:11.024519 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:11.024618 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:11.024625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.025323 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.025325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:11.025896 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:11.025904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.026318 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.026320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:11.026379 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:11.026906 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:11.026918 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:11.029189 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":68,"threads_started":1}
W20260808 04:51:11.029479 30181 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:11.023730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:55475 (local address 127.15.30.126:40593)
0808 04:51:11.023891 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:11.023895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:11.023983 (+ 88us) server_negotiation.cc:407] Connection header received
0808 04:51:11.024104 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:11.024107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:11.024159 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:11.024227 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:11.024718 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.025229 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.025996 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.026218 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.026301 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:11.029331 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:55475: BlockingRecv error: recv got EOF from 127.15.30.67:55475 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 04:51:11 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:11 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:11.666677 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:11.657702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:50665)
0808 04:51:11.657977 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:11.657992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:11.658083 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:11.658385 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:11.658390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:11.658639 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:11.658745 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:11.658752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.659542 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.659547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:11.660113 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:11.660122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.660614 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.660617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:11.660692 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:11.664674 (+ 3982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:11.664686 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:11.666557 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
W20260808 04:51:11.666855 30185 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:11.657786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:50665 (local address 127.15.30.126:40593)
0808 04:51:11.658029 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:11.658032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:11.658044 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:51:11.658148 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:11.658150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:11.658197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:11.658275 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:11.658867 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.659414 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.660228 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.660510 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.660592 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:11.666702 (+ 6110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:50665: BlockingRecv error: recv got EOF from 127.15.30.65:50665 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
Aug 08 04:51:11 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:11 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:11.851758 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:11.846124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:46723)
0808 04:51:11.846421 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:11.846438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:11.846581 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:11.846877 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:11.846879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:11.847099 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:11.847204 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:11.847210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.847977 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.847982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:11.848604 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:11.848613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.849089 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.849093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:11.849191 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:11.849764 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:11.849776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:11.851583 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
W20260808 04:51:11.851824 30185 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:11.846184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:46723 (local address 127.15.30.126:40593)
0808 04:51:11.846318 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:11.846323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:11.846529 (+ 206us) server_negotiation.cc:407] Connection header received
0808 04:51:11.846702 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:11.846705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:11.846758 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:11.846856 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:11.847329 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.847862 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.848734 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:11.848960 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:11.849084 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:11.851716 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:46723: BlockingRecv error: recv got EOF from 127.15.30.66:46723 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 04:51:12 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:12 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:12.035409 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:12.030109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:56707)
0808 04:51:12.030373 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:12.030389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:12.030495 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:12.030741 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:12.030744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:12.030922 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:12.031019 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:12.031024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.031767 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.031770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:12.032328 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:12.032336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.032804 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.032807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:12.032871 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:12.033401 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:12.033412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:12.035293 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
W20260808 04:51:12.035604 30185 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:12.030152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:56707 (local address 127.15.30.126:40593)
0808 04:51:12.030293 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:12.030296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:12.030446 (+ 150us) server_negotiation.cc:407] Connection header received
0808 04:51:12.030581 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:12.030584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:12.030627 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:12.030687 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:12.031131 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.031659 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.032485 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.032704 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.032787 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:12.035442 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:56707: BlockingRecv error: recv got EOF from 127.15.30.67:56707 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:51:12 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:12 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:12.673831 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:12.667695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:55563)
0808 04:51:12.667973 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:12.667986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:12.668112 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:12.668456 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:12.668459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:12.668654 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:12.668748 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:12.668754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.669526 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.669528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:12.670104 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:12.670113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.670606 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.670610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:12.670689 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:12.671334 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:12.671347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:12.673641 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
W20260808 04:51:12.674002 30196 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:12.667756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:55563 (local address 127.15.30.126:40593)
0808 04:51:12.668016 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:12.668019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:12.668034 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:51:12.668222 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:12.668225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:12.668274 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:12.668357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:12.668852 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.669410 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.670219 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.670457 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.670542 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:12.673829 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:55563: BlockingRecv error: recv got EOF from 127.15.30.65:55563 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
Aug 08 04:51:12 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:12 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:12.858314 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:12.852696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:60759)
0808 04:51:12.852981 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:12.852995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:12.853210 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:12.853473 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:12.853475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:12.853675 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:12.853770 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:12.853775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.854500 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.854503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:12.855066 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:12.855075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.855499 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.855502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:12.855573 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:12.856152 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:12.856165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:12.858197 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
W20260808 04:51:12.858484 30196 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:12.852812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:60759 (local address 127.15.30.126:40593)
0808 04:51:12.852979 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:12.852983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:12.853068 (+ 85us) server_negotiation.cc:407] Connection header received
0808 04:51:12.853322 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:12.853324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:12.853370 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:12.853437 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:12.853870 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.854378 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.855172 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:12.855380 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:12.855456 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:12.858328 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:60759: BlockingRecv error: recv got EOF from 127.15.30.66:60759 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 04:51:13 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:13 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:13.042241 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:13.036298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:56661)
0808 04:51:13.036747 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:13.036759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:13.036890 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:13.037172 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:13.037175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:13.037410 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:13.037503 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:13.037509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.038280 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.038285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:13.038870 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:13.038878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.039343 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.039347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:13.039438 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:13.040038 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:13.040054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:13.042121 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":127,"threads_started":1}
W20260808 04:51:13.042373 30196 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:13.036400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:56661 (local address 127.15.30.126:40593)
0808 04:51:13.036675 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:13.036680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:13.036839 (+ 159us) server_negotiation.cc:407] Connection header received
0808 04:51:13.037000 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:13.037003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:13.037059 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:13.037147 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:13.037628 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.038158 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.038996 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.039227 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.039338 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:13.042254 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:56661: BlockingRecv error: recv got EOF from 127.15.30.67:56661 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Aug 08 04:51:13 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:13 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:13.680275 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:13.674839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:60903)
0808 04:51:13.675111 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:13.675127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:13.675238 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:13.675567 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:13.675570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:13.675865 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:13.675986 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:13.675992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.676796 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.676798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:13.677341 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:13.677349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.677792 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.677795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:13.677853 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:13.678339 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:13.678351 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:13.680147 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260808 04:51:13.680624 30204 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:13.674855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:60903 (local address 127.15.30.126:40593)
0808 04:51:13.675172 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:13.675177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:13.675193 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:51:13.675343 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:13.675347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:13.675412 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:13.675524 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:13.676094 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.676698 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.677452 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.677694 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.677770 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:13.680398 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:60903: BlockingRecv error: recv got EOF from 127.15.30.65:60903 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
Aug 08 04:51:13 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:13 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:13.865049 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:13.859125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:47845)
0808 04:51:13.859419 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:13.859433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:13.859565 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:13.859885 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:13.859891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:13.860196 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:13.860354 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:13.860361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.861260 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.861263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:13.861848 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:13.861857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.862342 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.862345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:13.862412 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:13.862931 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:13.862943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:13.864931 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
W20260808 04:51:13.865386 30204 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:13.859216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:47845 (local address 127.15.30.126:40593)
0808 04:51:13.859353 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:13.859357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:13.859508 (+ 151us) server_negotiation.cc:407] Connection header received
0808 04:51:13.859660 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:13.859662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:13.859742 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:13.859828 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:13.860501 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.861104 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.861972 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:13.862196 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:13.862311 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:13.865104 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:47845: BlockingRecv error: recv got EOF from 127.15.30.66:47845 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 08 04:51:14 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:14 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:14.048504 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:14.043155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:33271)
0808 04:51:14.043424 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:14.043438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:14.043538 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:14.043841 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:14.043844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:14.044038 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:14.044138 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:14.044143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.044888 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.044890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:14.045445 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:14.045453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.045904 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.045907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:14.045971 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:14.046471 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:14.046482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:14.048386 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260808 04:51:14.048681 30204 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:14.043252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:33271 (local address 127.15.30.126:40593)
0808 04:51:14.043392 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:14.043396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:14.043509 (+ 113us) server_negotiation.cc:407] Connection header received
0808 04:51:14.043633 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:14.043635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:14.043686 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:14.043763 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:14.044244 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.044788 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.045566 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.045795 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.045877 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:14.048547 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:33271: BlockingRecv error: recv got EOF from 127.15.30.67:33271 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 04:51:14 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:14 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:14.687749 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:14.681150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:38411)
0808 04:51:14.681541 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:14.681564 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:14.681668 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:14.681912 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:14.681914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:14.682104 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:14.682226 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:14.682233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.683029 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.683032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:14.683603 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:14.683612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.684019 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.684021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:14.684080 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:14.685231 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:14.685246 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:14.687576 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
W20260808 04:51:14.687830 30209 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:14.681224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:38411 (local address 127.15.30.126:40593)
0808 04:51:14.681559 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:14.681563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:14.681618 (+ 55us) server_negotiation.cc:407] Connection header received
0808 04:51:14.681758 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:14.681760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:14.681805 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:14.681873 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:14.682364 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.682928 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.683709 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.683922 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.683997 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:14.687698 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:38411: BlockingRecv error: recv got EOF from 127.15.30.65:38411 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":143,"threads_started":1}
Aug 08 04:51:14 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:14 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:14.871687 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:14.865871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:55195)
0808 04:51:14.866213 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:14.866237 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:14.866343 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:14.866588 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:14.866590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:14.866797 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:14.866895 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:14.866901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.867694 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.867697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:14.868279 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:14.868287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.868912 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.868917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:14.869004 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:14.869702 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:14.869721 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:14.871566 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
W20260808 04:51:14.871968 30209 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:14.865916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:55195 (local address 127.15.30.126:40593)
0808 04:51:14.866057 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:14.866061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:14.866333 (+ 272us) server_negotiation.cc:407] Connection header received
0808 04:51:14.866424 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:14.866426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:14.866474 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:14.866551 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:14.867002 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.867558 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.868441 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:14.868779 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:14.868888 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:14.871752 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:55195: BlockingRecv error: recv got EOF from 127.15.30.66:55195 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 04:51:15 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:15 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:15.055459 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:15.049408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:40753)
0808 04:51:15.049775 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:15.049787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:15.049929 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:15.050220 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:15.050223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:15.050447 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:15.050539 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:15.050544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.051500 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.051504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:15.052315 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:15.052326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.052790 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.052793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:15.052857 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:15.053362 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:15.053374 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:15.055343 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":86,"threads_started":1}
W20260808 04:51:15.055622 30209 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:15.049473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:40753 (local address 127.15.30.126:40593)
0808 04:51:15.049652 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:15.049657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:15.049868 (+ 211us) server_negotiation.cc:407] Connection header received
0808 04:51:15.050031 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:15.050036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:15.050090 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:15.050194 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:15.050664 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.051363 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.052460 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.052689 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.052772 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:15.055472 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:40753: BlockingRecv error: recv got EOF from 127.15.30.67:40753 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:51:15 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:15 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:15.695366 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:15.688627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:35625)
0808 04:51:15.688994 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:15.689008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:15.689117 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:15.689402 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:15.689404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:15.689597 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:15.689700 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:15.689705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.691013 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.691017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:15.691727 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:15.691739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.692491 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.692495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:15.692585 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:15.693171 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:15.693183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:15.695250 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":94,"threads_started":1}
W20260808 04:51:15.695797 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:15.688815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35625 (local address 127.15.30.126:40593)
0808 04:51:15.689109 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:15.689115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:15.689134 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:51:15.689188 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:15.689193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:15.689260 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:15.689375 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:15.689828 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.690876 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.691851 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.692316 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.692477 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:15.695445 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:35625: BlockingRecv error: recv got EOF from 127.15.30.65:35625 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
Aug 08 04:51:15 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:15 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:15.878973 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:15.872555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:47265)
0808 04:51:15.872949 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:15.872963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:15.873065 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:15.873365 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:15.873367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:15.873579 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:15.873680 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:15.873686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.874680 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.874683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:15.875394 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:15.875406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.876009 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.876013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:15.876104 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:15.876754 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:15.876767 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:15.878852 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":92,"threads_started":1}
W20260808 04:51:15.879165 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:15.872847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:47265 (local address 127.15.30.126:40593)
0808 04:51:15.872978 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:15.872982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:15.873013 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:51:15.873181 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:15.873183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:15.873230 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:15.873299 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:15.873798 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.874557 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.875539 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:15.875854 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:15.875976 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:15.879004 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:47265: BlockingRecv error: recv got EOF from 127.15.30.66:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 04:51:16 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:16 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:16.062295 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:16.056265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:40635)
0808 04:51:16.056601 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:16.056615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:16.056734 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:16.056960 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:16.056962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:16.057149 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:16.057269 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:16.057274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.058076 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.058083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:16.058920 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:16.058932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.059621 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.059625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:16.059712 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:16.060177 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:16.060188 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:16.062139 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":143,"threads_started":1}
W20260808 04:51:16.062369 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:16.056310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:40635 (local address 127.15.30.126:40593)
0808 04:51:16.056481 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:16.056486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:16.056696 (+ 210us) server_negotiation.cc:407] Connection header received
0808 04:51:16.056804 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:16.056806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:16.056856 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:16.056923 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:16.057370 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.057960 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.059062 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.059432 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.059546 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:16.062276 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:40635: BlockingRecv error: recv got EOF from 127.15.30.67:40635 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 08 04:51:16 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:16 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:16.707726 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:16.696225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:39837)
0808 04:51:16.696569 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:16.696585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:16.696698 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:16.696939 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:16.696941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:16.697122 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:16.697225 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:16.697231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.697959 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.697962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:16.698522 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:16.698530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.698951 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.698954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:16.699014 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:16.705546 (+ 6532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:16.705558 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:16.707578 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
W20260808 04:51:16.707858 30222 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:16.696338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:39837 (local address 127.15.30.126:40593)
0808 04:51:16.696597 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:16.696601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:16.696663 (+ 62us) server_negotiation.cc:407] Connection header received
0808 04:51:16.696775 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:16.696778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:16.696825 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:16.696902 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:16.697326 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.697853 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.698627 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.698852 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.698929 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:16.707723 (+ 8794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:39837: BlockingRecv error: recv got EOF from 127.15.30.65:39837 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":61,"threads_started":1}
Aug 08 04:51:16 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:16 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:16.885462 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:16.879871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:46771)
0808 04:51:16.880184 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:16.880199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:16.880305 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:16.880605 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:16.880608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:16.880811 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:16.880966 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:16.880973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.881760 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.881763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:16.882399 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:16.882409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.882800 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.882802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:16.882862 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:16.883343 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:16.883354 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:16.885328 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
W20260808 04:51:16.885689 30222 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:16.879951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:46771 (local address 127.15.30.126:40593)
0808 04:51:16.880122 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:16.880126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:16.880300 (+ 174us) server_negotiation.cc:407] Connection header received
0808 04:51:16.880428 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:16.880432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:16.880491 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:16.880569 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:16.881076 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.881663 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.882488 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:16.882701 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:16.882781 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:16.885532 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:46771: BlockingRecv error: recv got EOF from 127.15.30.66:46771 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 08 04:51:17 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:17 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:17.068964 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:17.063295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:60799)
0808 04:51:17.063635 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:17.063649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:17.063769 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:17.064017 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:17.064019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:17.064218 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:17.064320 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:17.064325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.065102 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.065105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:17.065656 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:17.065663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.066100 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.066103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:17.066168 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:17.066669 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:17.066681 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:17.068765 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
W20260808 04:51:17.069005 30222 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:17.063464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:60799 (local address 127.15.30.126:40593)
0808 04:51:17.063619 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:17.063624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:17.063726 (+ 102us) server_negotiation.cc:407] Connection header received
0808 04:51:17.063852 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:17.063854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:17.063902 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:17.063972 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:17.064464 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.064999 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.065763 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.065981 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.066059 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:17.068897 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:60799: BlockingRecv error: recv got EOF from 127.15.30.67:60799 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 04:51:17 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:17 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:17.715185 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:17.708572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:54797)
0808 04:51:17.708936 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:17.708971 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:17.709103 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:17.709438 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:17.709441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:17.709674 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:17.709838 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:17.709846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.710828 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.710832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:17.711411 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:17.711421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.711930 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.711934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:17.712023 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:17.712739 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:17.712754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:17.714981 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":75,"threads_started":1}
W20260808 04:51:17.715639 30228 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:17.708599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:54797 (local address 127.15.30.126:40593)
0808 04:51:17.708808 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:17.708812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:17.709095 (+ 283us) server_negotiation.cc:407] Connection header received
0808 04:51:17.709217 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:17.709220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:17.709300 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:17.709398 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:17.709987 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.710715 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.711566 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.711804 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.711913 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:17.715479 (+ 3566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:54797: BlockingRecv error: recv got EOF from 127.15.30.65:54797 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
Aug 08 04:51:17 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:17 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:17.892057 30230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:17.886290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:36039)
0808 04:51:17.886579 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:17.886600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:17.886743 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:17.887020 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:17.887023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:17.887240 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:17.887336 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:17.887342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.888197 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.888200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:17.888806 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:17.888814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.889299 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.889302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:17.889388 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:17.889949 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:17.889961 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:17.891885 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
W20260808 04:51:17.892094 30228 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:17.886368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:36039 (local address 127.15.30.126:40593)
0808 04:51:17.886546 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:17.886550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:17.886677 (+ 127us) server_negotiation.cc:407] Connection header received
0808 04:51:17.886846 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:17.886850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:17.886907 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:17.886993 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:17.887469 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.888032 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.888931 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:17.889173 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:17.889285 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:17.892000 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:36039: BlockingRecv error: recv got EOF from 127.15.30.66:36039 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 04:51:18 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:18 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:18.075739 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:18.069856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:53143)
0808 04:51:18.070184 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:18.070198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:18.070299 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:18.070524 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:18.070526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:18.070724 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:18.070823 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:18.070829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.071575 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.071580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:18.072176 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:18.072186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.072802 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.072808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:18.072898 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:18.073610 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:18.073631 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:18.075621 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
W20260808 04:51:18.075899 30228 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:18.070010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:53143 (local address 127.15.30.126:40593)
0808 04:51:18.070164 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:18.070168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:18.070255 (+ 87us) server_negotiation.cc:407] Connection header received
0808 04:51:18.070368 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:18.070370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:18.070413 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:18.070482 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:18.070928 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.071450 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.072290 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.072641 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.072751 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:18.075776 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:53143: BlockingRecv error: recv got EOF from 127.15.30.67:53143 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 04:51:18 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:18 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:18.722275 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:18.716907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:59045)
0808 04:51:18.717290 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:18.717307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:18.717424 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:18.717664 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:18.717666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:18.717842 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:18.717938 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:18.717943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.718693 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.718696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:18.719252 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:18.719260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.719692 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.719695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:18.719756 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:18.720345 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:18.720360 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:18.722161 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":150,"threads_started":1}
W20260808 04:51:18.722420 30233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:18.716978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:59045 (local address 127.15.30.126:40593)
0808 04:51:18.717274 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:18.717280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:18.717380 (+ 100us) server_negotiation.cc:407] Connection header received
0808 04:51:18.717492 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:18.717494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:18.717539 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:18.717637 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:18.718039 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.718570 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.719363 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.719576 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.719661 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:18.722289 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:59045: BlockingRecv error: recv got EOF from 127.15.30.65:59045 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
Aug 08 04:51:18 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:18 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:18.898525 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:18.892881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:35555)
0808 04:51:18.893148 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:18.893161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:18.893257 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:18.893497 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:18.893499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:18.893688 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:18.893800 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:18.893805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.894645 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.894648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:18.895252 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:18.895261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.895708 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.895711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:18.895801 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:18.896395 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:18.896407 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:18.898384 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
W20260808 04:51:18.898602 30233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:18.892934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:35555 (local address 127.15.30.126:40593)
0808 04:51:18.893123 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:18.893127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:18.893228 (+ 101us) server_negotiation.cc:407] Connection header received
0808 04:51:18.893334 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:18.893337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:18.893382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:18.893464 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:18.893925 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.894538 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.895389 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:18.895611 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:18.895692 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:18.898510 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:35555: BlockingRecv error: recv got EOF from 127.15.30.66:35555 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 04:51:19 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:19 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:19.082491 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:19.076593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:59933)
0808 04:51:19.076913 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:19.076927 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:19.077026 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:19.077279 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:19.077281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:19.077494 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:19.077591 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:19.077596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.078408 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.078411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:19.078986 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:19.078994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.079447 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.079450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:19.079513 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:19.080109 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:19.080122 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:19.082328 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":113,"threads_started":1}
W20260808 04:51:19.082612 30233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:19.076706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:59933 (local address 127.15.30.126:40593)
0808 04:51:19.076872 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:19.076876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:19.076999 (+ 123us) server_negotiation.cc:407] Connection header received
0808 04:51:19.077109 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:19.077112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:19.077162 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:19.077235 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:19.077725 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.078274 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.079118 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.079339 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.079424 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:19.082484 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:59933: BlockingRecv error: recv got EOF from 127.15.30.67:59933 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 04:51:19 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:19 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:19.728696 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:19.723173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:33659)
0808 04:51:19.723484 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:19.723498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:19.723605 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:19.723864 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:19.723867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:19.724059 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:19.724165 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:19.724171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.724948 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.724951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:19.725524 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:19.725534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.726001 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.726004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:19.726067 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:19.726551 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:19.726563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:19.728522 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
W20260808 04:51:19.728798 30241 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:19.723295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:33659 (local address 127.15.30.126:40593)
0808 04:51:19.723530 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:19.723534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:19.723560 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:51:19.723698 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:19.723700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:19.723747 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:19.723826 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:19.724276 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.724845 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.725663 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.725891 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.725982 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:19.728663 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:33659: BlockingRecv error: recv got EOF from 127.15.30.65:33659 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
Aug 08 04:51:19 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:19 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:19.905184 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:19.899368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:48213)
0808 04:51:19.899662 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:19.899677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:19.899788 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:19.900042 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:19.900044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:19.900228 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:19.900329 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:19.900334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.901164 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.901168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:19.901735 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:19.901745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.902242 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.902245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:19.902314 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:19.902811 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:19.902822 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:19.905006 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
W20260808 04:51:19.905320 30241 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:19.899440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:48213 (local address 127.15.30.126:40593)
0808 04:51:19.899620 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:19.899624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:19.899775 (+ 151us) server_negotiation.cc:407] Connection header received
0808 04:51:19.899869 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:19.899872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:19.899923 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:19.899997 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:19.900485 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.901043 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.901874 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:19.902127 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:19.902220 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:19.905179 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:48213: BlockingRecv error: recv got EOF from 127.15.30.66:48213 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 04:51:20 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:20 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:20.089156 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:20.083326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:58353)
0808 04:51:20.083632 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:20.083645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:20.083790 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:20.084063 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:20.084066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:20.084295 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:20.084415 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:20.084421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.085240 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.085244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:20.085803 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:20.085811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.086291 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.086295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:20.086383 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:20.086894 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:20.086906 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:20.088976 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
W20260808 04:51:20.089294 30241 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:20.083401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:58353 (local address 127.15.30.126:40593)
0808 04:51:20.083569 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:20.083575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:20.083727 (+ 152us) server_negotiation.cc:407] Connection header received
0808 04:51:20.083894 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:20.083899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:20.083951 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:20.084036 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:20.084543 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.085089 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.085933 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.086166 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.086273 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:20.089164 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:58353: BlockingRecv error: recv got EOF from 127.15.30.67:58353 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 04:51:20 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:20 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:20.735167 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:20.729564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:33671)
0808 04:51:20.729884 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:20.729899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:20.730008 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:20.730285 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:20.730287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:20.730480 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:20.730584 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:20.730589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.731429 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.731432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:20.732013 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:20.732022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.732576 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.732579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:20.732647 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:20.733143 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:20.733154 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:20.735031 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
W20260808 04:51:20.735337 30246 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:20.729616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:33671 (local address 127.15.30.126:40593)
0808 04:51:20.729905 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:20.729908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:20.729975 (+ 67us) server_negotiation.cc:407] Connection header received
0808 04:51:20.730103 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:20.730106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:20.730152 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:20.730229 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:20.730731 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.731311 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.732207 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.732466 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.732550 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:20.735158 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:33671: BlockingRecv error: recv got EOF from 127.15.30.65:33671 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
Aug 08 04:51:20 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:20 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:20.911564 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:20.906136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:37089)
0808 04:51:20.906457 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:20.906472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:20.906577 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:20.906804 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:20.906807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:20.906990 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:20.907087 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:20.907092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.907790 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.907792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:20.908392 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:20.908425 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.908920 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.908924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:20.909014 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:20.909593 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:20.909605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:20.911449 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
W20260808 04:51:20.911839 30246 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:20.906319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:37089 (local address 127.15.30.126:40593)
0808 04:51:20.906481 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:20.906484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:20.906540 (+ 56us) server_negotiation.cc:407] Connection header received
0808 04:51:20.906661 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:20.906663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:20.906708 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:20.906777 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:20.907185 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.907697 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.908554 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:20.908799 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:20.908924 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:20.911586 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:37089: BlockingRecv error: recv got EOF from 127.15.30.66:37089 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 04:51:21 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:21 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:21.095690 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:21.090058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:45555)
0808 04:51:21.090337 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:21.090353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:21.090466 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:21.090721 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:21.090723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:21.090911 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:21.091007 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:21.091013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.091764 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.091766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:21.092321 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:21.092328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.092874 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.092877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:21.092942 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:21.093500 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:21.093512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:21.095513 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
W20260808 04:51:21.095824 30246 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:21.090130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:45555 (local address 127.15.30.126:40593)
0808 04:51:21.090326 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:21.090335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:21.090470 (+ 135us) server_negotiation.cc:407] Connection header received
0808 04:51:21.090542 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:21.090556 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:21.090608 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:21.090682 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:21.091124 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.091653 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.092452 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.092731 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.092830 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:21.095682 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:45555: BlockingRecv error: recv got EOF from 127.15.30.67:45555 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 08 04:51:21 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:21 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:21.745733 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:21.735995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:35907)
0808 04:51:21.736326 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:21.736343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:21.736478 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:21.736817 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:21.736820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:21.737012 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:21.737150 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:21.737156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.737949 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.737952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:21.738557 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:21.738566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.739007 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.739009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:21.739110 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:21.743420 (+ 4310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:21.743434 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:21.745600 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
W20260808 04:51:21.745949 30252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:21.736046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35907 (local address 127.15.30.126:40593)
0808 04:51:21.736351 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:21.736354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:21.736421 (+ 67us) server_negotiation.cc:407] Connection header received
0808 04:51:21.736635 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:21.736639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:21.736702 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:21.736793 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:21.737284 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.737833 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.738674 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.738901 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.738983 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:21.745766 (+ 6783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:35907: BlockingRecv error: recv got EOF from 127.15.30.65:35907 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":64,"threads_started":1}
Aug 08 04:51:21 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:21 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:21.917879 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:21.912343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:58213)
0808 04:51:21.912660 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:21.912673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:21.912829 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:21.913114 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:21.913117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:21.913342 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:21.913436 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:21.913442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.914211 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.914216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:21.914805 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:21.914814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.915275 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.915278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:21.915361 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:21.915842 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:21.915854 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:21.917679 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
W20260808 04:51:21.917900 30252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:21.912424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:58213 (local address 127.15.30.126:40593)
0808 04:51:21.912600 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:21.912604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:21.912753 (+ 149us) server_negotiation.cc:407] Connection header received
0808 04:51:21.912935 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:21.912939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:21.912994 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:21.913087 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:21.913567 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.914095 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.914933 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:21.915159 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:21.915279 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:21.917801 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:58213: BlockingRecv error: recv got EOF from 127.15.30.66:58213 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 04:51:22 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:22 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.67@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:22.102334 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:22.096562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.67:35777)
0808 04:51:22.096857 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:22.096871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:22.096973 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:22.097259 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:22.097262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:22.097458 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:22.097552 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:22.097558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.098370 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.098373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:22.098955 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:22.098963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.099438 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.099441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:22.099506 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:22.100107 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:22.100122 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:22.102141 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
W20260808 04:51:22.102375 30252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:22.096680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:35777 (local address 127.15.30.126:40593)
0808 04:51:22.096822 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:22.096825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:22.096939 (+ 114us) server_negotiation.cc:407] Connection header received
0808 04:51:22.097089 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:22.097091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:22.097141 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:22.097215 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:22.097672 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.098235 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.099094 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.099308 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.099390 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:22.102269 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.67:35777: BlockingRecv error: recv got EOF from 127.15.30.67:35777 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 04:51:22 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:22 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.65@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:22.752561 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:22.746655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.65:56287)
0808 04:51:22.746982 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:22.746996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:22.747078 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:22.747365 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:22.747367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:22.747556 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:22.747655 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:22.747660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.748486 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.748489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:22.749071 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:22.749119 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.749577 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.749580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:22.749643 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:22.750174 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:22.750186 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:22.752390 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
W20260808 04:51:22.752707 30256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:22.746770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:56287 (local address 127.15.30.126:40593)
0808 04:51:22.747019 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:22.747023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:22.747054 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:51:22.747202 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:22.747205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:22.747253 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:22.747326 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:22.747772 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.748328 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.749245 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.749471 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.749572 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:22.752562 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.65:56287: BlockingRecv error: recv got EOF from 127.15.30.65:56287 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":68,"threads_started":1}
Aug 08 04:51:22 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:51:22 dist-test-slave-zpvx krb5kdc[28625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.30.66@KRBTEST.COM for oryx/127.15.30.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:51:22.924345 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:22.918792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40593 (local address 127.15.30.66:55737)
0808 04:51:22.919065 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:22.919078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:22.919178 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:22.919429 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:22.919432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:22.919641 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:22.919737 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:22.919742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.920563 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.920566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:22.921124 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:22.921134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.921598 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.921600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:22.921667 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:22.922334 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:22.922345 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:22.924219 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.30.126:40593: Server oryx/127.15.30.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
W20260808 04:51:22.924562 30256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:22.918994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:55737 (local address 127.15.30.126:40593)
0808 04:51:22.919140 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:22.919149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:22.919163 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:51:22.919261 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:22.919263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:22.919307 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:22.919379 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:22.919850 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.920447 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.921256 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:22.921490 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:22.921589 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:22.924385 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.30.66:55737: BlockingRecv error: recv got EOF from 127.15.30.66:55737 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:51:22.993613 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 29531
I20260808 04:51:23.004540 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 29765
I20260808 04:51:23.017580 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 29952
I20260808 04:51:23.029383 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 29427
2026-08-08T04:51:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18273 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T04:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:51:23Z Disabled control of system clock
I20260808 04:51:23.065230 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:38583
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:38583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:51:23.176780 30264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:23.177022 30264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:23.177055 30264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:23.180907 30264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:51:23.180954 30264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:23.180969 30264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:23.180985 30264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:51:23.181000 30264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:51:23.185511 30264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:38583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:38583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:23.186430 30264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:23.187698 30264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:23.196727 30271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:23.196745 30264 server_base.cc:1034] running on GCE node
W20260808 04:51:23.196761 30272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:23.196745 30275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:23.197475 30264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:23.198549 30264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:23.199754 30264 hybrid_clock.cc:648] HybridClock initialized: now 1786164683199735 us; error 32 us; skew 500 ppm
I20260808 04:51:23.201792 30264 webserver.cc:460] Webserver started at http://127.15.30.126:40735/ using document root <none> and password file <none>
I20260808 04:51:23.202373 30264 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:23.202469 30264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:23.202713 30264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:23.204795 30264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "28dee52eca224fd996afe17dfe88b155"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "3c044c1b311d7cd0cb4de60cd2d8fd26"
server_key_iv: "5e9a76a6746078c6e464072dc3c59bd3"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.205360 30264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "28dee52eca224fd996afe17dfe88b155"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "3c044c1b311d7cd0cb4de60cd2d8fd26"
server_key_iv: "5e9a76a6746078c6e464072dc3c59bd3"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.208928 30264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:51:23.211424 30282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.212723 30264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:51:23.212865 30264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "28dee52eca224fd996afe17dfe88b155"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "3c044c1b311d7cd0cb4de60cd2d8fd26"
server_key_iv: "5e9a76a6746078c6e464072dc3c59bd3"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.212977 30264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:23.231312 30264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:23.231940 30264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:23.232131 30264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:23.239341 30264 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:38583
I20260808 04:51:23.239369 30360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:38583 every 8 connection(s)
I20260808 04:51:23.240263 30264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:51:23.240911 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 30264
I20260808 04:51:23.241058 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:51:23.241408 15481 external_mini_cluster.cc:1428] Setting key 162e66311b3756fae167cc26f8f2d70c
I20260808 04:51:23.244211 30361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:23.251040 30361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155: Bootstrap starting.
I20260808 04:51:23.251875 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:23.242769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.15.30.126:38583)
0808 04:51:23.243441 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:23.243461 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:23.243506 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:51:23.244418 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:23.244442 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:23.244935 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:23.245300 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:23.245743 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.247201 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.248097 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.248807 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.248996 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:23.249837 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:23.249843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:23.249859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:23.250359 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:23.250430 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:23.250457 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:23.250558 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:23.250724 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 04:51:23.250977 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260808 04:51:23.253566 30361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:23.254415 30361 log.cc:824] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:23.256228 30361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155: No bootstrap required, opened a new log
I20260808 04:51:23.259743 30361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dee52eca224fd996afe17dfe88b155" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38583 } }
I20260808 04:51:23.259979 30361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:23.260068 30361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dee52eca224fd996afe17dfe88b155, State: Initialized, Role: FOLLOWER
I20260808 04:51:23.260545 30361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dee52eca224fd996afe17dfe88b155" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38583 } }
I20260808 04:51:23.260695 30361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:51:23.260780 30361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:51:23.260879 30361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:23.261754 30361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dee52eca224fd996afe17dfe88b155" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38583 } }
I20260808 04:51:23.262089 30361 leader_election.cc:302] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28dee52eca224fd996afe17dfe88b155; no voters:
I20260808 04:51:23.262400 30361 leader_election.cc:288] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:51:23.262564 30370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:23.262847 30370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [term 1 LEADER]: Becoming Leader. State: Replica: 28dee52eca224fd996afe17dfe88b155, State: Running, Role: LEADER
I20260808 04:51:23.263195 30370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dee52eca224fd996afe17dfe88b155" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38583 } }
I20260808 04:51:23.263680 30361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:51:23.265082 30373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28dee52eca224fd996afe17dfe88b155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dee52eca224fd996afe17dfe88b155" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38583 } } }
I20260808 04:51:23.265039 30374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28dee52eca224fd996afe17dfe88b155. Latest consensus state: current_term: 1 leader_uuid: "28dee52eca224fd996afe17dfe88b155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28dee52eca224fd996afe17dfe88b155" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 38583 } } }
I20260808 04:51:23.265228 30373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:23.265249 30374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155 [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:23.265712 30390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:51:23.268433 30390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:51:23.274011 30390 catalog_manager.cc:1347] Generated new cluster ID: b1dbd86627d04494bfd22e991c2e63f6
I20260808 04:51:23.274116 30390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:51:23.296789 30390 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:51:23.297778 30390 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:51:23.302732 30390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28dee52eca224fd996afe17dfe88b155: Generated new TSK 0
I20260808 04:51:23.303418 30390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:51:23.314549 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:38583
--builtin_ntp_servers=127.15.30.84:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 04:51:23.428781 30405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:23.429042 30405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:23.429105 30405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:23.433152 30405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:23.433197 30405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:23.433280 30405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:51:23.437878 30405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:23.438922 30405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:23.440295 30405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:23.450105 30411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:23.450114 30413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:23.450316 30415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:23.450394 30405 server_base.cc:1034] running on GCE node
I20260808 04:51:23.450811 30405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:23.451475 30405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:23.452630 30405 hybrid_clock.cc:648] HybridClock initialized: now 1786164683452600 us; error 43 us; skew 500 ppm
I20260808 04:51:23.454397 30405 webserver.cc:460] Webserver started at http://127.15.30.65:41755/ using document root <none> and password file <none>
I20260808 04:51:23.455013 30405 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:23.455080 30405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:23.455291 30405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:23.457377 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "bd5f8d9ee3644b99aefe33b207c4d1d8"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "63426ef9aec5677a76bbc6d5b0624947"
server_key_iv: "f22700fd5ecd1ddd34dfda95c22dcb3a"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.457944 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "bd5f8d9ee3644b99aefe33b207c4d1d8"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "63426ef9aec5677a76bbc6d5b0624947"
server_key_iv: "f22700fd5ecd1ddd34dfda95c22dcb3a"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.461642 30405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:51:23.464138 30425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.465525 30405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:51:23.465912 30405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "bd5f8d9ee3644b99aefe33b207c4d1d8"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "63426ef9aec5677a76bbc6d5b0624947"
server_key_iv: "f22700fd5ecd1ddd34dfda95c22dcb3a"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.466028 30405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:23.476841 30405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:23.477490 30405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:23.477684 30405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:23.478267 30405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:23.479189 30405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:23.479238 30405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.479301 30405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:23.479333 30405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.488786 30405 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:39201
I20260808 04:51:23.488811 30594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:39201 every 8 connection(s)
I20260808 04:51:23.489699 30405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:51:23.491909 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 30405
I20260808 04:51:23.492004 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:51:23.492333 15481 external_mini_cluster.cc:1428] Setting key 496844d384ef4d505c91ecff9a48636d
I20260808 04:51:23.494295 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:38583
--builtin_ntp_servers=127.15.30.84:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:51:23.499621 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:23.491529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:33987 (local address 127.15.30.126:38583)
0808 04:51:23.491703 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:23.491708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:23.492948 (+ 1240us) server_negotiation.cc:407] Connection header received
0808 04:51:23.494386 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:23.494389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:23.494444 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:23.494525 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:23.495419 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.496088 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.497310 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.497700 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.497827 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:23.498974 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:23.498980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:23.498987 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:23.499005 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:23.499033 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:23.499046 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:23.499129 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:23.499294 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 04:51:23.499395 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:51:23.500317 30599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:23.492025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:38583 (local address 127.15.30.65:33987)
0808 04:51:23.492800 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:23.492852 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:23.494173 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:23.494698 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:23.494705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:23.494787 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:23.495270 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:23.495282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.496246 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.496249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:23.497169 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:23.497180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.497855 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.497858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:23.498001 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:23.498661 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:51:23.498662 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:23.498703 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:51:23.498766 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:23.499149 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:23.499157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:23.499162 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:51:23.499255 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:51:23.499470 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20260808 04:51:23.501665 30595 heartbeater.cc:347] Connected to a master server at 127.15.30.126:38583
I20260808 04:51:23.501890 30595 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:23.502405 30595 heartbeater.cc:511] Master 127.15.30.126:38583 requested a full tablet report, sending...
I20260808 04:51:23.503908 30301 ts_manager.cc:194] Registered new tserver with Master: bd5f8d9ee3644b99aefe33b207c4d1d8 (127.15.30.65:39201)
I20260808 04:51:23.505156 30301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.65:33987
W20260808 04:51:23.606671 30600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:23.606902 30600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:23.606938 30600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:23.610939 30600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:23.610987 30600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:23.611074 30600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:51:23.615705 30600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:23.616801 30600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:23.618321 30600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:23.627836 30606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:23.628085 30610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:23.627938 30600 server_base.cc:1034] running on GCE node
W20260808 04:51:23.627876 30608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:23.628772 30600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:23.629449 30600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:23.630616 30600 hybrid_clock.cc:648] HybridClock initialized: now 1786164683630582 us; error 47 us; skew 500 ppm
I20260808 04:51:23.632531 30600 webserver.cc:460] Webserver started at http://127.15.30.66:42473/ using document root <none> and password file <none>
I20260808 04:51:23.633109 30600 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:23.633170 30600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:23.633394 30600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:23.635427 30600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "725bd8fd24dd475eb47095daa46e14b0"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "4add9faaf0ff8ab712e6900a119978bc"
server_key_iv: "1b8ab6c6874aab87757b375c3ea66d5c"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.635998 30600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "725bd8fd24dd475eb47095daa46e14b0"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "4add9faaf0ff8ab712e6900a119978bc"
server_key_iv: "1b8ab6c6874aab87757b375c3ea66d5c"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.639734 30600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 04:51:23.642354 30618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.643631 30600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:23.643769 30600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "725bd8fd24dd475eb47095daa46e14b0"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "4add9faaf0ff8ab712e6900a119978bc"
server_key_iv: "1b8ab6c6874aab87757b375c3ea66d5c"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.643875 30600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:23.676335 30600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:23.677157 30600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:23.677398 30600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:23.678038 30600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:23.679015 30600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:23.679062 30600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.679129 30600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:23.679163 30600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.688225 30600 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:33849
I20260808 04:51:23.688266 30789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:33849 every 8 connection(s)
I20260808 04:51:23.689199 30600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:51:23.691936 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 30600
I20260808 04:51:23.692059 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:51:23.692420 15481 external_mini_cluster.cc:1428] Setting key 60f7b580dad5a09d38ccba203bb35296
I20260808 04:51:23.694396 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:38583
--builtin_ntp_servers=127.15.30.84:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:51:23.699507 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:23.691175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:40811 (local address 127.15.30.126:38583)
0808 04:51:23.691358 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:23.691363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:23.692529 (+ 1166us) server_negotiation.cc:407] Connection header received
0808 04:51:23.693918 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:23.693922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:23.693976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:23.694061 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:23.695037 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.695565 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.697275 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.697633 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.697762 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:23.698875 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:23.698882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:23.698898 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:23.698916 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:23.698943 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:23.698957 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:23.699036 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:23.699203 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:51:23.699283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:51:23.700268 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:23.691582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:38583 (local address 127.15.30.66:40811)
0808 04:51:23.692319 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:23.692405 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:23.693703 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:23.694253 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:23.694260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:23.694345 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:23.694889 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:23.694902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.695695 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.695699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:23.697110 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:23.697125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.697819 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.697822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:23.697963 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:23.698591 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:51:23.698593 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:23.698633 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:51:23.698696 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:23.699045 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:23.699053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:23.699057 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:51:23.699155 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:51:23.699381 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":140,"threads_started":1}
I20260808 04:51:23.701476 30792 heartbeater.cc:347] Connected to a master server at 127.15.30.126:38583
I20260808 04:51:23.701757 30792 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:23.702438 30792 heartbeater.cc:511] Master 127.15.30.126:38583 requested a full tablet report, sending...
I20260808 04:51:23.703660 30301 ts_manager.cc:194] Registered new tserver with Master: 725bd8fd24dd475eb47095daa46e14b0 (127.15.30.66:33849)
I20260808 04:51:23.704523 30301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.66:40811
W20260808 04:51:23.810807 30798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:23.811074 30798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:23.811139 30798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:23.815116 30798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:23.815192 30798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:23.815274 30798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:51:23.819911 30798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:23.821008 30798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:23.822328 30798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:23.831960 30804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:23.831949 30805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:23.831955 30807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:23.832244 30798 server_base.cc:1034] running on GCE node
I20260808 04:51:23.832806 30798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:23.833595 30798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:23.834739 30798 hybrid_clock.cc:648] HybridClock initialized: now 1786164683834690 us; error 45 us; skew 500 ppm
I20260808 04:51:23.836987 30798 webserver.cc:460] Webserver started at http://127.15.30.67:39241/ using document root <none> and password file <none>
I20260808 04:51:23.837577 30798 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:23.837641 30798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:23.837863 30798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:23.839905 30798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "a6641ee42bd44b5eb81e0f1001676962"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "00ecbd91b4e3d02e00f4c309ba225c23"
server_key_iv: "8b0000b438f399a7604afe673b5dc2ba"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.840516 30798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "a6641ee42bd44b5eb81e0f1001676962"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "00ecbd91b4e3d02e00f4c309ba225c23"
server_key_iv: "8b0000b438f399a7604afe673b5dc2ba"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.844089 30798 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:51:23.846616 30817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.847751 30798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:23.847898 30798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "a6641ee42bd44b5eb81e0f1001676962"
format_stamp: "Formatted at 2026-08-08 04:51:23 on dist-test-slave-zpvx"
server_key: "00ecbd91b4e3d02e00f4c309ba225c23"
server_key_iv: "8b0000b438f399a7604afe673b5dc2ba"
server_key_version: "encryptionkey@0"
I20260808 04:51:23.848006 30798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:23.872092 30798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:23.872771 30798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:23.872964 30798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:23.873544 30798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:23.874475 30798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:23.874553 30798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.874622 30798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:23.874665 30798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:23.883895 30798 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:46711
I20260808 04:51:23.883952 30994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:46711 every 8 connection(s)
I20260808 04:51:23.884824 30798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:51:23.892468 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 30798
I20260808 04:51:23.892599 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:51:23.892884 15481 external_mini_cluster.cc:1428] Setting key 2ac697bb9ec9fa042adee92390087609
I20260808 04:51:23.894616 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:23.886887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:36921 (local address 127.15.30.126:38583)
0808 04:51:23.887084 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:23.887088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:23.888143 (+ 1055us) server_negotiation.cc:407] Connection header received
0808 04:51:23.889673 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:23.889677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:23.889733 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:23.889854 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:23.890726 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.891264 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.892481 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.892800 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.892916 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:23.894082 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:23.894086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:23.894089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:23.894102 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:23.894160 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:23.894189 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:23.894268 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:23.894458 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 04:51:23.894506 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:51:23.895556 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:23.887322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:38583 (local address 127.15.30.67:36921)
0808 04:51:23.887986 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:23.888025 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:23.889451 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:23.890017 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:23.890025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:23.890113 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:23.890572 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:23.890584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.891396 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.891405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:23.892311 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:23.892322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:23.892956 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:23.892960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:23.893131 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:23.893808 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:51:23.893810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:23.893866 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:51:23.893928 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:23.894273 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:23.894281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:23.894287 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:51:23.894448 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 04:51:23.894718 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260808 04:51:23.896668 30995 heartbeater.cc:347] Connected to a master server at 127.15.30.126:38583
I20260808 04:51:23.896872 30995 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:23.897333 30995 heartbeater.cc:511] Master 127.15.30.126:38583 requested a full tablet report, sending...
I20260808 04:51:23.898288 30301 ts_manager.cc:194] Registered new tserver with Master: a6641ee42bd44b5eb81e0f1001676962 (127.15.30.67:46711)
I20260808 04:51:23.898809 30301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.67:36921
I20260808 04:51:23.906896 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:51:23.909627 30368 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:23.908849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.15.30.126:38583)
0808 04:51:23.908962 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:23.908966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:23.909152 (+ 186us) server_negotiation.cc:407] Connection header received
0808 04:51:23.909248 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:23.909251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:23.909288 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:23.909352 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:23.909530 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33596: BlockingRecv error: recv got EOF from 127.0.0.1:33596 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:51:23.909832 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 30405
I20260808 04:51:23.925412 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 30600
I20260808 04:51:23.940083 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 30798
I20260808 04:51:23.949250 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 30264
2026-08-08T04:51:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (918 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T04:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:51:23Z Disabled control of system clock
I20260808 04:51:23.983366 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:40453
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:51:24.094785 31012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:24.095022 31012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:24.095057 31012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:24.098927 31012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:51:24.098977 31012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:24.098992 31012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:24.099009 31012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:51:24.099023 31012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:51:24.103587 31012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:40453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:24.104496 31012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:24.105762 31012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:51:24.114070 31012 server_base.cc:1034] running on GCE node
W20260808 04:51:24.114032 31023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:24.114028 31025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:24.114323 31021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:24.114924 31012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:24.115957 31012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:24.117125 31012 hybrid_clock.cc:648] HybridClock initialized: now 1786164684117106 us; error 35 us; skew 500 ppm
I20260808 04:51:24.118878 31012 webserver.cc:460] Webserver started at http://127.15.30.126:35557/ using document root <none> and password file <none>
I20260808 04:51:24.119395 31012 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:24.119454 31012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:24.119626 31012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:24.121675 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "b4e5820f200441e9992eebc7a48151ac"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "6b1b9858706a53dab060c793b286f17f"
server_key_iv: "53ebc2f561c0de78ef7839262f79a664"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.122179 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "b4e5820f200441e9992eebc7a48151ac"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "6b1b9858706a53dab060c793b286f17f"
server_key_iv: "53ebc2f561c0de78ef7839262f79a664"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.125681 31012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:51:24.127923 31039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.129281 31012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:24.129417 31012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "b4e5820f200441e9992eebc7a48151ac"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "6b1b9858706a53dab060c793b286f17f"
server_key_iv: "53ebc2f561c0de78ef7839262f79a664"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.129524 31012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:24.133839 31012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:24.134413 31012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:24.134590 31012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:24.141460 31012 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:40453
I20260808 04:51:24.141489 31121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:40453 every 8 connection(s)
I20260808 04:51:24.142318 31012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:51:24.145560 31123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:24.148926 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 31012
I20260808 04:51:24.149078 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:51:24.149359 15481 external_mini_cluster.cc:1428] Setting key 4131b2725a4079f09a4aedb998acdb55
I20260808 04:51:24.151574 31123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac: Bootstrap starting.
I20260808 04:51:24.154199 31123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:24.155035 31123 log.cc:824] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:24.155593 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:24.150908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.15.30.126:40453)
0808 04:51:24.151576 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:24.151593 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:24.151664 (+ 71us) server_negotiation.cc:407] Connection header received
0808 04:51:24.152481 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:24.152510 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:24.152889 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:24.153192 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:24.154091 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:24.154112 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:24.154125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:24.154139 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:24.154198 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:24.154216 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:24.154370 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:24.154512 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 04:51:24.154821 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":163,"threads_started":1}
I20260808 04:51:24.157094 31123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac: No bootstrap required, opened a new log
I20260808 04:51:24.160831 31123 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5820f200441e9992eebc7a48151ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40453 } }
I20260808 04:51:24.161041 31123 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:24.161074 31123 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e5820f200441e9992eebc7a48151ac, State: Initialized, Role: FOLLOWER
I20260808 04:51:24.161485 31123 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5820f200441e9992eebc7a48151ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40453 } }
I20260808 04:51:24.161680 31123 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:51:24.161772 31123 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:51:24.161892 31123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:24.162825 31123 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5820f200441e9992eebc7a48151ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40453 } }
I20260808 04:51:24.163193 31123 leader_election.cc:302] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e5820f200441e9992eebc7a48151ac; no voters:
I20260808 04:51:24.163492 31123 leader_election.cc:288] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:51:24.163666 31130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:24.163926 31130 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [term 1 LEADER]: Becoming Leader. State: Replica: b4e5820f200441e9992eebc7a48151ac, State: Running, Role: LEADER
I20260808 04:51:24.164280 31130 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5820f200441e9992eebc7a48151ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40453 } }
I20260808 04:51:24.164762 31123 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:51:24.166088 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e5820f200441e9992eebc7a48151ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5820f200441e9992eebc7a48151ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40453 } } }
I20260808 04:51:24.166096 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e5820f200441e9992eebc7a48151ac. Latest consensus state: current_term: 1 leader_uuid: "b4e5820f200441e9992eebc7a48151ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5820f200441e9992eebc7a48151ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 40453 } } }
I20260808 04:51:24.166239 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:24.166272 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:24.167133 31141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:51:24.170745 31141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:51:24.176350 31141 catalog_manager.cc:1347] Generated new cluster ID: 0b3e0869c46c4beb8c12f25ed2ba4982
I20260808 04:51:24.176438 31141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:51:24.188023 31141 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:51:24.188316 31141 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:51:24.192814 31141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4e5820f200441e9992eebc7a48151ac: Generated new TSK 0
I20260808 04:51:24.193441 31141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:51:24.202078 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40453
--builtin_ntp_servers=127.15.30.84:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:51:24.317106 31159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:24.317341 31159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:24.317379 31159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:24.321342 31159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:24.321391 31159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:24.321471 31159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:51:24.326195 31159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:24.327220 31159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:24.328665 31159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:24.338639 31176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:24.338639 31170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:24.338822 31159 server_base.cc:1034] running on GCE node
W20260808 04:51:24.338646 31172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:24.339450 31159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:24.340070 31159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:24.341265 31159 hybrid_clock.cc:648] HybridClock initialized: now 1786164684341227 us; error 50 us; skew 500 ppm
I20260808 04:51:24.343117 31159 webserver.cc:460] Webserver started at http://127.15.30.65:40431/ using document root <none> and password file <none>
I20260808 04:51:24.343704 31159 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:24.343802 31159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:24.344036 31159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:24.346112 31159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "20b3ce120fa841f3b94a7923244f2a42"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "75fec3f1ec2f60f211716b72ac86d28b"
server_key_iv: "69938473d01603033cd110699ca0bc14"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.346684 31159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "20b3ce120fa841f3b94a7923244f2a42"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "75fec3f1ec2f60f211716b72ac86d28b"
server_key_iv: "69938473d01603033cd110699ca0bc14"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.350852 31159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:51:24.353483 31184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.354627 31159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:51:24.354759 31159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "20b3ce120fa841f3b94a7923244f2a42"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "75fec3f1ec2f60f211716b72ac86d28b"
server_key_iv: "69938473d01603033cd110699ca0bc14"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.354879 31159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:24.359360 31159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:24.359948 31159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:24.360136 31159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:24.360755 31159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:24.361649 31159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:24.361721 31159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.361792 31159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:24.361837 31159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.371083 31159 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:46331
I20260808 04:51:24.371129 31353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:46331 every 8 connection(s)
I20260808 04:51:24.371949 31159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:51:24.378479 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:24.373815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:46621 (local address 127.15.30.126:40453)
0808 04:51:24.373950 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:24.373954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:24.374951 (+ 997us) server_negotiation.cc:407] Connection header received
0808 04:51:24.376554 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:24.376557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:24.376610 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:24.376721 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:24.377965 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:24.377970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:24.377973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:24.377987 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:24.378016 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:24.378025 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:24.378130 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:24.378295 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 04:51:24.378342 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:51:24.379298 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:24.374130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40453 (local address 127.15.30.65:46621)
0808 04:51:24.374793 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:24.374834 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:24.376264 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:24.376903 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:24.376910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:24.377027 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:24.377695 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:51:24.377696 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:24.377737 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:51:24.377805 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:24.378124 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:24.378134 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:24.378142 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:51:24.378327 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 04:51:24.378565 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260808 04:51:24.379470 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 31159
I20260808 04:51:24.379608 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:51:24.379936 15481 external_mini_cluster.cc:1428] Setting key 5fd4e9dbc6054ad83b5b415886acf8a1
I20260808 04:51:24.380518 31354 heartbeater.cc:347] Connected to a master server at 127.15.30.126:40453
I20260808 04:51:24.380787 31354 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:24.381222 31354 heartbeater.cc:511] Master 127.15.30.126:40453 requested a full tablet report, sending...
I20260808 04:51:24.382225 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40453
--builtin_ntp_servers=127.15.30.84:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 04:51:24.382786 31071 ts_manager.cc:194] Registered new tserver with Master: 20b3ce120fa841f3b94a7923244f2a42 (127.15.30.65:46331)
W20260808 04:51:24.498672 31357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:24.498921 31357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:24.498961 31357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:24.502910 31357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:24.502964 31357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:24.503015 31357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:51:24.507532 31357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:24.508517 31357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:24.509845 31357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:24.519266 31367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:24.519302 31362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:24.519383 31357 server_base.cc:1034] running on GCE node
W20260808 04:51:24.519304 31363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:24.519927 31357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:24.520656 31357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:24.521857 31357 hybrid_clock.cc:648] HybridClock initialized: now 1786164684521836 us; error 30 us; skew 500 ppm
I20260808 04:51:24.523716 31357 webserver.cc:460] Webserver started at http://127.15.30.66:39571/ using document root <none> and password file <none>
I20260808 04:51:24.524334 31357 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:24.524482 31357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:24.524724 31357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:24.526744 31357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "306bd9ecfea04222a7b3d366c3028b6f"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "ec20ade0b4a4047b67cba7e241ad11e6"
server_key_iv: "98f656e0275e4112ab8472654b0f12ea"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.527307 31357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "306bd9ecfea04222a7b3d366c3028b6f"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "ec20ade0b4a4047b67cba7e241ad11e6"
server_key_iv: "98f656e0275e4112ab8472654b0f12ea"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.531167 31357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:51:24.533878 31380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.535034 31357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:51:24.535213 31357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "306bd9ecfea04222a7b3d366c3028b6f"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "ec20ade0b4a4047b67cba7e241ad11e6"
server_key_iv: "98f656e0275e4112ab8472654b0f12ea"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.535329 31357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:24.539836 31357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:24.540488 31357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:24.540668 31357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:24.541211 31357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:24.542115 31357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:24.542193 31357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.542260 31357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:24.542305 31357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.551023 31357 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:41667
I20260808 04:51:24.551051 31543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:41667 every 8 connection(s)
I20260808 04:51:24.551877 31357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:51:24.558136 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:24.553754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:34053 (local address 127.15.30.126:40453)
0808 04:51:24.553919 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:24.553924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:24.555021 (+ 1097us) server_negotiation.cc:407] Connection header received
0808 04:51:24.556302 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:24.556305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:24.556344 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:24.556511 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:24.557566 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:24.557569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:24.557571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:24.557584 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:24.557607 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:24.557615 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:24.557760 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:24.557980 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 04:51:24.558022 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:51:24.559020 31549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:24.554194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40453 (local address 127.15.30.66:34053)
0808 04:51:24.554870 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:24.554910 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:24.556083 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:24.556710 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:24.556717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:24.556803 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:24.557312 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:51:24.557314 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:24.557351 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:51:24.557419 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:24.557774 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:24.557784 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:24.557789 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:51:24.557944 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 04:51:24.558175 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260808 04:51:24.559633 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 31357
I20260808 04:51:24.559757 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:51:24.560096 15481 external_mini_cluster.cc:1428] Setting key c60a87ca9e8e2e514de18dc86b873bcc
I20260808 04:51:24.560354 31545 heartbeater.cc:347] Connected to a master server at 127.15.30.126:40453
I20260808 04:51:24.560624 31545 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:24.561064 31545 heartbeater.cc:511] Master 127.15.30.126:40453 requested a full tablet report, sending...
I20260808 04:51:24.562181 31071 ts_manager.cc:194] Registered new tserver with Master: 306bd9ecfea04222a7b3d366c3028b6f (127.15.30.66:41667)
I20260808 04:51:24.562202 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40453
--builtin_ntp_servers=127.15.30.84:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:51:24.676563 31550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:24.676800 31550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:24.676836 31550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:24.680837 31550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:24.680886 31550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:24.680966 31550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:51:24.686902 31550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:40453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:24.688210 31550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:24.689996 31550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:24.700673 31562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:24.700673 31560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:24.700698 31564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:24.700927 31550 server_base.cc:1034] running on GCE node
I20260808 04:51:24.701319 31550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:24.701925 31550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:24.703155 31550 hybrid_clock.cc:648] HybridClock initialized: now 1786164684703139 us; error 26 us; skew 500 ppm
I20260808 04:51:24.705075 31550 webserver.cc:460] Webserver started at http://127.15.30.67:41291/ using document root <none> and password file <none>
I20260808 04:51:24.705665 31550 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:24.705761 31550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:24.705996 31550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:24.708043 31550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "8a91fe87a75b463f8f7cb713e7a0a3bd"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "41ee73cd260ebdfa4435c26dedfb7105"
server_key_iv: "ea373c1fefa6e86fc07f02a4c21c1ee5"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.708657 31550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "8a91fe87a75b463f8f7cb713e7a0a3bd"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "41ee73cd260ebdfa4435c26dedfb7105"
server_key_iv: "ea373c1fefa6e86fc07f02a4c21c1ee5"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.712343 31550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:51:24.714821 31574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.716097 31550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:24.716240 31550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "8a91fe87a75b463f8f7cb713e7a0a3bd"
format_stamp: "Formatted at 2026-08-08 04:51:24 on dist-test-slave-zpvx"
server_key: "41ee73cd260ebdfa4435c26dedfb7105"
server_key_iv: "ea373c1fefa6e86fc07f02a4c21c1ee5"
server_key_version: "encryptionkey@0"
I20260808 04:51:24.716353 31550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:24.720827 31550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:24.721428 31550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:24.721613 31550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:24.722142 31550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:24.723019 31550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:24.723099 31550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.723170 31550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:24.723220 31550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:24.731859 31550 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:45659
I20260808 04:51:24.731910 31747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:45659 every 8 connection(s)
I20260808 04:51:24.732844 31550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:51:24.739320 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 31550
I20260808 04:51:24.739418 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:24.734803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:58349 (local address 127.15.30.126:40453)
0808 04:51:24.734942 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:24.734946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:24.736092 (+ 1146us) server_negotiation.cc:407] Connection header received
0808 04:51:24.737469 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:24.737472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:24.737512 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:24.737591 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:24.738823 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:24.738826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:24.738829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:51:24.738841 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:24.738863 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:51:24.738871 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:24.738961 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:24.739224 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 04:51:24.739275 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:51:24.739475 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:51:24.739768 15481 external_mini_cluster.cc:1428] Setting key 6bc459e70c2497d06e1fe847c7d15b2f
I20260808 04:51:24.740584 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:24.735253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:40453 (local address 127.15.30.67:58349)
0808 04:51:24.735942 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:24.735982 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:24.737261 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:24.737761 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:24.737768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:24.737854 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:24.738542 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:51:24.738543 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:24.738585 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:51:24.738660 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:24.738986 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:24.738995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:24.739000 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:51:24.739265 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 04:51:24.739670 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
I20260808 04:51:24.741786 31748 heartbeater.cc:347] Connected to a master server at 127.15.30.126:40453
I20260808 04:51:24.742022 31748 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:24.742461 31748 heartbeater.cc:511] Master 127.15.30.126:40453 requested a full tablet report, sending...
I20260808 04:51:24.743536 31071 ts_manager.cc:194] Registered new tserver with Master: 8a91fe87a75b463f8f7cb713e7a0a3bd (127.15.30.67:45659)
I20260808 04:51:24.754297 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:51:24.757349 31128 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:51:24.756297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.15.30.126:40453)
0808 04:51:24.756505 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:24.756510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:24.756657 (+ 147us) server_negotiation.cc:407] Connection header received
0808 04:51:24.756815 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:24.756818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:24.756864 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:24.756952 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:24.757216 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35102: BlockingRecv error: recv got EOF from 127.0.0.1:35102 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 04:51:24.757589 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 31159
I20260808 04:51:24.777266 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 31357
I20260808 04:51:24.789717 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 31550
I20260808 04:51:24.801661 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 31012
2026-08-08T04:51:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (852 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:51:24 dist-test-slave-zpvx krb5kdc[31761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:51:24 dist-test-slave-zpvx krb5kdc[31761](info): set up 2 sockets
Aug 08 04:51:24 dist-test-slave-zpvx krb5kdc[31761](info): commencing operation
krb5kdc: starting...
W20260808 04:51:26.059156 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:51:26 dist-test-slave-zpvx krb5kdc[31761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:51:26.187050 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42847
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:42847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:26.301198 31790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:26.301515 31790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:26.301592 31790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:26.305562 31790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:51:26.305649 31790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:26.305685 31790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:26.305725 31790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:51:26.305756 31790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:51:26.310912 31790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:42847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:26.311915 31790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:26.312934 31790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:26.321823 31801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:26.321843 31799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:26.321943 31790 server_base.cc:1034] running on GCE node
W20260808 04:51:26.321848 31804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:26.322680 31790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:26.323809 31790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:26.325027 31790 hybrid_clock.cc:648] HybridClock initialized: now 1786164686324996 us; error 43 us; skew 500 ppm
Aug 08 04:51:26 dist-test-slave-zpvx krb5kdc[31761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:26.327907 31790 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:51:26.328919 31790 webserver.cc:460] Webserver started at http://127.15.30.126:44071/ using document root <none> and password file <none>
I20260808 04:51:26.329480 31790 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:26.329571 31790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:26.329787 31790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:26.331892 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "c72a326243144e7b8f868637dcd7d1ac"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "4884fde4772a7f8e3261484f75dd0a61"
server_key_iv: "4fe178ed56a5dc41db583b63db45e544"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.332522 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "c72a326243144e7b8f868637dcd7d1ac"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "4884fde4772a7f8e3261484f75dd0a61"
server_key_iv: "4fe178ed56a5dc41db583b63db45e544"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.336299 31790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:51:26.338666 31814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:26.339815 31790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:26.339943 31790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "c72a326243144e7b8f868637dcd7d1ac"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "4884fde4772a7f8e3261484f75dd0a61"
server_key_iv: "4fe178ed56a5dc41db583b63db45e544"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.340039 31790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:26.359359 31790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:26.362639 31790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:26.362845 31790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:26.369900 31790 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:42847
I20260808 04:51:26.370179 31884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:42847 every 8 connection(s)
I20260808 04:51:26.370882 31790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:51:26.373251 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 31790
I20260808 04:51:26.373386 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:51:26.373744 15481 external_mini_cluster.cc:1428] Setting key 62aed7ce5d0055a4184b62655ff7204b
I20260808 04:51:26.373939 31886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:26.380942 31886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac: Bootstrap starting.
Aug 08 04:51:26 dist-test-slave-zpvx krb5kdc[31761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:51:26.383647 31886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:26.384443 31886 log.cc:824] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:26.386415 31886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac: No bootstrap required, opened a new log
I20260808 04:51:26.388846 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:26.375151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.15.30.126:42847)
0808 04:51:26.375561 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:26.375577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:26.375611 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:51:26.376669 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:26.376702 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:26.377017 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:26.377288 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:26.378211 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.379173 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.380326 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.380824 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.380940 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:26.383692 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:26.383697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:26.383710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:26.383737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:26.385769 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:26.386296 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:26.386306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:26.386311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:26.386437 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:26.386800 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:26.386803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:26.386805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:26.387241 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:26.387466 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:26.387787 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 04:51:26.388077 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
I20260808 04:51:26.390129 31886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a326243144e7b8f868637dcd7d1ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42847 } }
I20260808 04:51:26.390405 31886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:26.390484 31886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72a326243144e7b8f868637dcd7d1ac, State: Initialized, Role: FOLLOWER
I20260808 04:51:26.390924 31886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a326243144e7b8f868637dcd7d1ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42847 } }
I20260808 04:51:26.391101 31886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:51:26.391170 31886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:51:26.391305 31886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:26.392167 31886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a326243144e7b8f868637dcd7d1ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42847 } }
I20260808 04:51:26.392556 31886 leader_election.cc:302] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c72a326243144e7b8f868637dcd7d1ac; no voters:
I20260808 04:51:26.392875 31886 leader_election.cc:288] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:51:26.393014 31891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:26.393289 31891 raft_consensus.cc:686] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [term 1 LEADER]: Becoming Leader. State: Replica: c72a326243144e7b8f868637dcd7d1ac, State: Running, Role: LEADER
I20260808 04:51:26.393590 31891 consensus_queue.cc:237] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a326243144e7b8f868637dcd7d1ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42847 } }
I20260808 04:51:26.394096 31886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:51:26.395397 31895 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader c72a326243144e7b8f868637dcd7d1ac. Latest consensus state: current_term: 1 leader_uuid: "c72a326243144e7b8f868637dcd7d1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a326243144e7b8f868637dcd7d1ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42847 } } }
I20260808 04:51:26.395853 31895 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [sys.catalog]: This master's current role is: LEADER
I20260808 04:51:26.395736 31892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c72a326243144e7b8f868637dcd7d1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a326243144e7b8f868637dcd7d1ac" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42847 } } }
I20260808 04:51:26.396652 31892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac [sys.catalog]: This master's current role is: LEADER
W20260808 04:51:26.399091 31913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:51:26.399185 31913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:51:26.399247 31911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:51:26.399937 31911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:51:26.405464 31911 catalog_manager.cc:1347] Generated new cluster ID: 4c501b3ffee5477da50c7f569d6b725e
I20260808 04:51:26.405527 31911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:51:26.414481 31911 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:51:26.415668 31911 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:51:26.423740 31911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac: Generated new TSK 0
I20260808 04:51:26.424437 31911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:51:26.464288 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42847
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:26.595255 31925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:26.595582 31925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:26.595703 31925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:26.600276 31925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:26.600339 31925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:26.600473 31925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:51:26.605997 31925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:26.607077 31925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:26.608116 31925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:26.618037 31937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:26.618037 31934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:26.618206 31925 server_base.cc:1034] running on GCE node
W20260808 04:51:26.618395 31939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:26.618814 31925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:26.619515 31925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:26.620702 31925 hybrid_clock.cc:648] HybridClock initialized: now 1786164686620667 us; error 44 us; skew 500 ppm
Aug 08 04:51:26 dist-test-slave-zpvx krb5kdc[31761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:26.623814 31925 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:51:26.624830 31925 webserver.cc:460] Webserver started at http://127.15.30.65:37573/ using document root <none> and password file <none>
I20260808 04:51:26.625427 31925 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:26.625521 31925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:26.625773 31925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:26.627800 31925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "5e4fd13f45324405a35538ec3e42b51f"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "c3d686a3ee6e35ff31dc3a1587785149"
server_key_iv: "a95de67eb841783a2e9538f42e952c36"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.628443 31925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "5e4fd13f45324405a35538ec3e42b51f"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "c3d686a3ee6e35ff31dc3a1587785149"
server_key_iv: "a95de67eb841783a2e9538f42e952c36"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.632341 31925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:51:26.635004 31947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:26.636322 31925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:51:26.636500 31925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "5e4fd13f45324405a35538ec3e42b51f"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "c3d686a3ee6e35ff31dc3a1587785149"
server_key_iv: "a95de67eb841783a2e9538f42e952c36"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.636636 31925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:26.648227 31925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:26.651082 31925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:26.651286 31925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:26.651911 31925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:26.652812 31925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:26.652889 31925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:26.652958 31925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:26.653008 31925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:26.662147 31925 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:46371
I20260808 04:51:26.662195 32119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:46371 every 8 connection(s)
I20260808 04:51:26.663053 31925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:51:26.672295 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 31925
I20260808 04:51:26.672503 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:51:26.672849 15481 external_mini_cluster.cc:1428] Setting key e9fcac89c4441fd51bf6103fad527b63
Aug 08 04:51:26 dist-test-slave-zpvx krb5kdc[31761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:51:26.677821 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:26.665245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:49707 (local address 127.15.30.126:42847)
0808 04:51:26.665393 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:26.665397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:26.666256 (+ 859us) server_negotiation.cc:407] Connection header received
0808 04:51:26.667663 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:26.667666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:26.667724 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:26.667812 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:26.669010 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.669565 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.670738 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.670976 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.671092 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:26.673708 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:26.673712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:26.673715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:26.673743 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:26.675570 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:26.676075 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:26.676082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:26.676086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:26.676190 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:26.676594 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:26.676599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:26.676601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:26.676852 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:26.676969 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:26.677596 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 04:51:26.677703 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:51:26.678853 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:26.665623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42847 (local address 127.15.30.65:49707)
0808 04:51:26.666092 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:26.666136 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:26.667446 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:26.667996 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:26.668004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:26.668359 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:26.668854 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:26.668868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.669706 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.669715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:26.670574 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:26.670584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.671134 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.671138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:26.671309 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:26.671858 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:26.671873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:26.673549 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:26.675693 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:26.675698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:26.675708 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:26.675965 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:26.676299 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:26.676301 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:26.676303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:26.676499 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:26.676986 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:26.676995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:26.677309 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 04:51:26.677522 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 04:51:26.677882 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260808 04:51:26.680085 32120 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42847
I20260808 04:51:26.680306 32120 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:26.680799 32120 heartbeater.cc:511] Master 127.15.30.126:42847 requested a full tablet report, sending...
I20260808 04:51:26.682245 31834 ts_manager.cc:194] Registered new tserver with Master: 5e4fd13f45324405a35538ec3e42b51f (127.15.30.65:46371)
I20260808 04:51:26.683331 31834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:49707
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:51:26.705389 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42847
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:26.817054 32128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:26.817292 32128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:26.817333 32128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:26.821273 32128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:26.821326 32128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:26.821381 32128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:51:26.826453 32128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:26.827384 32128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:26.828341 32128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:51:26.837989 32135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:26.837996 32133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:26.838001 32137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:26.838460 32128 server_base.cc:1034] running on GCE node
I20260808 04:51:26.838836 32128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:26.839546 32128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:26.840724 32128 hybrid_clock.cc:648] HybridClock initialized: now 1786164686840708 us; error 43 us; skew 500 ppm
Aug 08 04:51:26 dist-test-slave-zpvx krb5kdc[31761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:26.848752 32128 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:51:26.849752 32128 webserver.cc:460] Webserver started at http://127.15.30.66:43171/ using document root <none> and password file <none>
I20260808 04:51:26.850364 32128 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:26.850461 32128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:26.850684 32128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:26.852844 32128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "342c7ce1d9ea41388535ff9881351735"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "cfacb815b942a384c57a676b71295140"
server_key_iv: "f067d437574d3250a158e3a51bde2354"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.853423 32128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "342c7ce1d9ea41388535ff9881351735"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "cfacb815b942a384c57a676b71295140"
server_key_iv: "f067d437574d3250a158e3a51bde2354"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.857188 32128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:51:26.859683 32148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:26.861119 32128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:51:26.861286 32128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "342c7ce1d9ea41388535ff9881351735"
format_stamp: "Formatted at 2026-08-08 04:51:26 on dist-test-slave-zpvx"
server_key: "cfacb815b942a384c57a676b71295140"
server_key_iv: "f067d437574d3250a158e3a51bde2354"
server_key_version: "encryptionkey@0"
I20260808 04:51:26.861406 32128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:26.874634 32128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:26.877666 32128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:26.877878 32128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:26.878486 32128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:26.879325 32128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:26.879398 32128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:26.879467 32128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:26.879518 32128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:26.888751 32128 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:34161
I20260808 04:51:26.888782 32312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:34161 every 8 connection(s)
I20260808 04:51:26.889648 32128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:51:26.891705 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 32128
I20260808 04:51:26.891819 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:51:26.892164 15481 external_mini_cluster.cc:1428] Setting key e586923f936889aeef504d415b037b6a
Aug 08 04:51:26 dist-test-slave-zpvx krb5kdc[31761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:51:26.904633 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:26.891483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:56159 (local address 127.15.30.126:42847)
0808 04:51:26.891664 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:26.891670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:26.892608 (+ 938us) server_negotiation.cc:407] Connection header received
0808 04:51:26.894146 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:26.894149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:26.894224 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:26.894348 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:26.895672 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.896273 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.897756 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.898003 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.898118 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:26.900820 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:26.900823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:26.900825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:26.900848 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:26.902317 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:26.902944 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:26.902946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:26.902948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:26.903065 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:26.903462 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:26.903465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:26.903466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:26.903645 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:26.903731 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:26.904340 (+ 609us) server_negotiation.cc:299] Negotiation successful
0808 04:51:26.904505 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:51:26.905468 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:26.891896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42847 (local address 127.15.30.66:56159)
0808 04:51:26.892435 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:26.892495 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:26.893909 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:26.894564 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:26.894571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:26.894922 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:26.895519 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:26.895535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.896636 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.896639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:26.897623 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:26.897633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:26.898158 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:26.898171 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:26.898341 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:26.898939 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:26.898956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:26.900643 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:26.902447 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:26.902456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:26.902475 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:26.902835 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:26.903157 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:26.903162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:26.903165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:26.903366 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:26.903752 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:26.903758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:26.903995 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 04:51:26.904255 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 04:51:26.904587 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
I20260808 04:51:26.906970 32313 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42847
I20260808 04:51:26.907279 32313 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:26.907830 32313 heartbeater.cc:511] Master 127.15.30.126:42847 requested a full tablet report, sending...
I20260808 04:51:26.908973 31834 ts_manager.cc:194] Registered new tserver with Master: 342c7ce1d9ea41388535ff9881351735 (127.15.30.66:34161)
I20260808 04:51:26.909586 31834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:56159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:51:26.922808 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42847
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:51:27.034085 32319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:51:27.034403 32319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:51:27.034468 32319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:51:27.038537 32319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:51:27.038602 32319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:51:27.038657 32319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:51:27.043845 32319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:51:27.044848 32319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:51:27.045784 32319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:51:27.055459 32319 server_base.cc:1034] running on GCE node
W20260808 04:51:27.055358 32326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:27.055426 32328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:51:27.055364 32330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:51:27.055953 32319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:51:27.056653 32319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:51:27.057852 32319 hybrid_clock.cc:648] HybridClock initialized: now 1786164687057840 us; error 26 us; skew 500 ppm
Aug 08 04:51:27 dist-test-slave-zpvx krb5kdc[31761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:51:27.060577 32319 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:51:27.061390 32319 webserver.cc:460] Webserver started at http://127.15.30.67:35937/ using document root <none> and password file <none>
I20260808 04:51:27.061942 32319 fs_manager.cc:359] Metadata directory not provided
I20260808 04:51:27.062034 32319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:51:27.062276 32319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:51:27.064251 32319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "7a0883807a964d0b8cc01f7008cc7124"
format_stamp: "Formatted at 2026-08-08 04:51:27 on dist-test-slave-zpvx"
server_key: "c4a216ab36ca2cff7017210d95cf6c95"
server_key_iv: "a7e7a6b1815d3328587ef41a24a19757"
server_key_version: "encryptionkey@0"
I20260808 04:51:27.064898 32319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "7a0883807a964d0b8cc01f7008cc7124"
format_stamp: "Formatted at 2026-08-08 04:51:27 on dist-test-slave-zpvx"
server_key: "c4a216ab36ca2cff7017210d95cf6c95"
server_key_iv: "a7e7a6b1815d3328587ef41a24a19757"
server_key_version: "encryptionkey@0"
I20260808 04:51:27.068547 32319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:51:27.070899 32340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:27.072082 32319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:51:27.072222 32319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "7a0883807a964d0b8cc01f7008cc7124"
format_stamp: "Formatted at 2026-08-08 04:51:27 on dist-test-slave-zpvx"
server_key: "c4a216ab36ca2cff7017210d95cf6c95"
server_key_iv: "a7e7a6b1815d3328587ef41a24a19757"
server_key_version: "encryptionkey@0"
I20260808 04:51:27.072330 32319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:51:27.097427 32319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:51:27.100554 32319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:51:27.100760 32319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:51:27.101356 32319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:51:27.102183 32319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:51:27.102254 32319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:27.102327 32319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:51:27.102380 32319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:51:27.111496 32319 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:42419
I20260808 04:51:27.111531 32519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:42419 every 8 connection(s)
I20260808 04:51:27.112358 32319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:51:27.119566 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 32319
I20260808 04:51:27.119706 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:51:27.119990 15481 external_mini_cluster.cc:1428] Setting key ee883c811ce006d55a3d0b27bfe546bf
Aug 08 04:51:27 dist-test-slave-zpvx krb5kdc[31761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:51:27.126466 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.114298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:45837 (local address 127.15.30.126:42847)
0808 04:51:27.114460 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.114464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.115297 (+ 833us) server_negotiation.cc:407] Connection header received
0808 04:51:27.116807 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.116810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.116859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.116936 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:27.118114 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.118683 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.119928 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.120229 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.120342 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.122956 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:27.122959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:27.122961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:27.122985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:27.124326 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:27.125030 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:27.125032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:27.125034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:27.125152 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:27.125490 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:27.125493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:27.125494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:27.125687 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:27.125778 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.126222 (+ 444us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.126348 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:51:27.127324 32522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.114646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42847 (local address 127.15.30.67:45837)
0808 04:51:27.115104 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:27.115147 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:27.116595 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:27.117133 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:27.117139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:27.117498 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:51:27.117970 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.117983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.118803 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.118806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.119808 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.119818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.120411 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.120416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.120585 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.121186 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:51:27.121202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:51:27.122808 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:51:27.124641 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:27.124649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:27.124666 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:27.124931 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:27.125241 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:51:27.125244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:51:27.125245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:51:27.125370 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:51:27.125791 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:51:27.125796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:51:27.126005 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 04:51:27.126165 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 04:51:27.126455 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260808 04:51:27.128505 32520 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42847
I20260808 04:51:27.128743 32520 heartbeater.cc:464] Registering TS with master...
I20260808 04:51:27.129231 32520 heartbeater.cc:511] Master 127.15.30.126:42847 requested a full tablet report, sending...
I20260808 04:51:27.130259 31834 ts_manager.cc:194] Registered new tserver with Master: 7a0883807a964d0b8cc01f7008cc7124 (127.15.30.67:42419)
I20260808 04:51:27.130820 31834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:45837
I20260808 04:51:27.133808 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:51:27.142720 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.135749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.15.30.126:42847)
0808 04:51:27.135883 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.135888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.136026 (+ 138us) server_negotiation.cc:407] Connection header received
0808 04:51:27.136117 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.136119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.136157 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.136219 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:51:27.136977 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.137449 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.138255 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.138451 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.138523 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.139841 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:51:27.139844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:51:27.139845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:51:27.139868 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:51:27.141270 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:27.141661 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:27.141663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:27.141664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:27.141736 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:51:27.141961 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:51:27.141963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:51:27.141964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:51:27.142152 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:51:27.142250 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.142446 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.142529 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:51:27.145754 31834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:51:27.148658 31834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:51:27.165537 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.158176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.15.30.67:42419)
0808 04:51:27.159679 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.159696 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.159721 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:51:27.159800 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.159807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.160024 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.160193 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:27.160498 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.161749 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.163582 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.164611 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.164743 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.164857 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.165251 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.165351 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":155,"threads_started":1}
I20260808 04:51:27.165530 32535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.158001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:42419 (local address 127.0.0.1:37028)
0808 04:51:27.158778 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:27.158792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:27.158882 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:27.160167 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:27.160173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:27.160194 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:27.160320 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.160327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.161875 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.161880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.163429 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.163441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.164909 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.164912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.165061 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.165089 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:51:27.165198 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:51:27.165376 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":173,"thread_start_us":92,"threads_started":1}
I20260808 04:51:27.165805 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.159432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.15.30.65:46371)
0808 04:51:27.159706 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.159729 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.160003 (+ 274us) server_negotiation.cc:407] Connection header received
0808 04:51:27.160117 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.160125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.160321 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.160493 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:27.160727 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.161861 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.163577 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.164514 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.164623 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.164788 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.165459 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.165649 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
I20260808 04:51:27.165995 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.159241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46371 (local address 127.0.0.1:49448)
0808 04:51:27.159821 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:27.159834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:27.160003 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:27.160488 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:27.160490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:27.160507 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:27.160607 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.160613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.162023 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.162026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.163429 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.163441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.164853 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.164856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.165016 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.165086 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 04:51:27.165196 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:51:27.165376 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":109,"thread_start_us":86,"threads_started":1}
I20260808 04:51:27.167891 32534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.157819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:34161 (local address 127.0.0.1:41652)
0808 04:51:27.158304 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:27.158348 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:27.158470 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:27.162648 (+ 4178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:27.162653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:27.162678 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:27.162803 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.162813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.164200 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.164205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.165351 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.165362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.167495 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.167498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.167607 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.167631 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:51:27.167711 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:51:27.167777 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":86,"threads_started":1}
I20260808 04:51:27.167981 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.158755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.15.30.66:34161)
0808 04:51:27.162090 (+ 3335us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.162104 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.162127 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:51:27.162216 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.162221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.162454 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.162972 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:27.163154 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.164062 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.165500 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.167358 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.167475 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.167604 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.167751 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.167835 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3211,"thread_start_us":127,"threads_started":1}
I20260808 04:51:27.168741 32414 tablet_service.cc:1467] Processing CreateTablet for tablet 35124e3f3e944a93bde84f1faaeec976 (DEFAULT_TABLE table=test-table [id=f25636fe55754b92a1a620acfd4cb297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:27.168716 32015 tablet_service.cc:1467] Processing CreateTablet for tablet 35124e3f3e944a93bde84f1faaeec976 (DEFAULT_TABLE table=test-table [id=f25636fe55754b92a1a620acfd4cb297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:27.169672 32015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35124e3f3e944a93bde84f1faaeec976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:27.169682 32414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35124e3f3e944a93bde84f1faaeec976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:27.175511 32540 tablet_bootstrap.cc:492] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Bootstrap starting.
I20260808 04:51:27.175190 32215 tablet_service.cc:1467] Processing CreateTablet for tablet 35124e3f3e944a93bde84f1faaeec976 (DEFAULT_TABLE table=test-table [id=f25636fe55754b92a1a620acfd4cb297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:51:27.176210 32215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35124e3f3e944a93bde84f1faaeec976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:51:27.176719 32542 tablet_bootstrap.cc:492] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f: Bootstrap starting.
I20260808 04:51:27.178373 32540 tablet_bootstrap.cc:654] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:27.178757 32542 tablet_bootstrap.cc:654] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:27.179502 32540 log.cc:824] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:27.180193 32542 log.cc:824] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:27.181445 32540 tablet_bootstrap.cc:492] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: No bootstrap required, opened a new log
I20260808 04:51:27.181658 32540 ts_tablet_manager.cc:1397] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:51:27.182147 32542 tablet_bootstrap.cc:492] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f: No bootstrap required, opened a new log
I20260808 04:51:27.182331 32542 ts_tablet_manager.cc:1397] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:51:27.183606 32545 tablet_bootstrap.cc:492] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Bootstrap starting.
I20260808 04:51:27.185376 32542 raft_consensus.cc:348] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.185576 32542 raft_consensus.cc:374] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:27.185614 32542 raft_consensus.cc:729] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4fd13f45324405a35538ec3e42b51f, State: Initialized, Role: FOLLOWER
I20260808 04:51:27.185557 32540 raft_consensus.cc:348] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.185817 32540 raft_consensus.cc:374] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:27.185890 32540 raft_consensus.cc:729] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0883807a964d0b8cc01f7008cc7124, State: Initialized, Role: FOLLOWER
I20260808 04:51:27.186036 32542 consensus_queue.cc:260] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.186455 32545 tablet_bootstrap.cc:654] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Neither blocks nor log segments found. Creating new log.
I20260808 04:51:27.186390 32540 consensus_queue.cc:260] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.186750 32120 heartbeater.cc:503] Master 127.15.30.126:42847 was elected leader, sending a full tablet report...
I20260808 04:51:27.187274 32545 log.cc:824] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Log is configured to *not* fsync() on all Append() calls
I20260808 04:51:27.187368 32540 ts_tablet_manager.cc:1428] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:51:27.187690 32520 heartbeater.cc:503] Master 127.15.30.126:42847 was elected leader, sending a full tablet report...
I20260808 04:51:27.187964 32542 ts_tablet_manager.cc:1428] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:51:27.189855 32545 tablet_bootstrap.cc:492] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: No bootstrap required, opened a new log
I20260808 04:51:27.190037 32545 ts_tablet_manager.cc:1397] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260808 04:51:27.193310 32545 raft_consensus.cc:348] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.193527 32545 raft_consensus.cc:374] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:51:27.193586 32545 raft_consensus.cc:729] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342c7ce1d9ea41388535ff9881351735, State: Initialized, Role: FOLLOWER
I20260808 04:51:27.193981 32545 consensus_queue.cc:260] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.194638 32313 heartbeater.cc:503] Master 127.15.30.126:42847 was elected leader, sending a full tablet report...
I20260808 04:51:27.194872 32545 ts_tablet_manager.cc:1428] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260808 04:51:27.303510 32548 raft_consensus.cc:482] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:51:27.303764 32548 raft_consensus.cc:504] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.304958 32548 leader_election.cc:288] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342c7ce1d9ea41388535ff9881351735 (127.15.30.66:34161), 7a0883807a964d0b8cc01f7008cc7124 (127.15.30.67:42419)
I20260808 04:51:27.309069 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.305331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:57937 (local address 127.15.30.66:34161)
0808 04:51:27.305483 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.305488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.305770 (+ 282us) server_negotiation.cc:407] Connection header received
0808 04:51:27.305943 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.305946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.305992 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.306079 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:27.306410 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.306908 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.308045 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.308483 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.308618 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.308674 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.308857 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.308926 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:51:27.309091 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.305228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:34161 (local address 127.15.30.65:57937)
0808 04:51:27.305654 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:27.305666 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:27.305808 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:27.306091 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:27.306094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:27.306109 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:27.306289 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.306295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.307024 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.307029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.307908 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.307918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.308619 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.308622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.308723 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.308751 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:51:27.308840 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:51:27.308909 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":38,"thread_start_us":152,"threads_started":1}
I20260808 04:51:27.309088 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.305395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:60189 (local address 127.15.30.67:42419)
0808 04:51:27.305524 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.305530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.305839 (+ 309us) server_negotiation.cc:407] Connection header received
0808 04:51:27.305991 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.305994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.306046 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.306168 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:51:27.306456 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.306947 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.308041 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.308513 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.308639 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.308692 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.308887 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.308954 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:51:27.309122 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.305234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:42419 (local address 127.15.30.65:60189)
0808 04:51:27.305730 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:27.305742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:51:27.305871 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:51:27.306148 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:51:27.306152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:51:27.306179 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:51:27.306340 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.306346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.307064 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.307070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.307911 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:51:27.307920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.308643 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.308647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:51:27.308762 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.308779 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 04:51:27.308869 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:51:27.308939 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":81,"spinlock_wait_cycles":896,"thread_start_us":123,"threads_started":1}
I20260808 04:51:27.309719 32246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35124e3f3e944a93bde84f1faaeec976" candidate_uuid: "5e4fd13f45324405a35538ec3e42b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342c7ce1d9ea41388535ff9881351735" is_pre_election: true
I20260808 04:51:27.309830 32446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35124e3f3e944a93bde84f1faaeec976" candidate_uuid: "5e4fd13f45324405a35538ec3e42b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0883807a964d0b8cc01f7008cc7124" is_pre_election: true
I20260808 04:51:27.310050 32246 raft_consensus.cc:2454] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4fd13f45324405a35538ec3e42b51f in term 0.
I20260808 04:51:27.310070 32446 raft_consensus.cc:2454] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4fd13f45324405a35538ec3e42b51f in term 0.
I20260808 04:51:27.310561 31952 leader_election.cc:302] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 342c7ce1d9ea41388535ff9881351735, 5e4fd13f45324405a35538ec3e42b51f, 7a0883807a964d0b8cc01f7008cc7124; no voters:
I20260808 04:51:27.310851 32548 raft_consensus.cc:2781] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:51:27.310926 32548 raft_consensus.cc:482] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:51:27.310956 32548 raft_consensus.cc:3037] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:27.312126 32548 raft_consensus.cc:504] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.312544 32548 leader_election.cc:288] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [CANDIDATE]: Term 1 election: Requested vote from peers 342c7ce1d9ea41388535ff9881351735 (127.15.30.66:34161), 7a0883807a964d0b8cc01f7008cc7124 (127.15.30.67:42419)
I20260808 04:51:27.312897 32246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35124e3f3e944a93bde84f1faaeec976" candidate_uuid: "5e4fd13f45324405a35538ec3e42b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342c7ce1d9ea41388535ff9881351735"
I20260808 04:51:27.312933 32446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35124e3f3e944a93bde84f1faaeec976" candidate_uuid: "5e4fd13f45324405a35538ec3e42b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0883807a964d0b8cc01f7008cc7124"
I20260808 04:51:27.312999 32246 raft_consensus.cc:3037] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:27.313081 32446 raft_consensus.cc:3037] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:51:27.314083 32246 raft_consensus.cc:2454] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4fd13f45324405a35538ec3e42b51f in term 1.
I20260808 04:51:27.314222 32446 raft_consensus.cc:2454] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4fd13f45324405a35538ec3e42b51f in term 1.
I20260808 04:51:27.314389 31952 leader_election.cc:302] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342c7ce1d9ea41388535ff9881351735, 5e4fd13f45324405a35538ec3e42b51f; no voters:
I20260808 04:51:27.314596 32548 raft_consensus.cc:2781] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:51:27.314796 32548 raft_consensus.cc:686] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 1 LEADER]: Becoming Leader. State: Replica: 5e4fd13f45324405a35538ec3e42b51f, State: Running, Role: LEADER
I20260808 04:51:27.315110 32548 consensus_queue.cc:237] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } }
I20260808 04:51:27.318120 31834 catalog_manager.cc:5495] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e4fd13f45324405a35538ec3e42b51f (127.15.30.65). New cstate: current_term: 1 leader_uuid: "5e4fd13f45324405a35538ec3e42b51f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4fd13f45324405a35538ec3e42b51f" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46371 } health_report { overall_health: HEALTHY } } }
I20260808 04:51:27.354801 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.351626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.15.30.65:46371)
0808 04:51:27.351786 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:51:27.351790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:51:27.351803 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:51:27.351872 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:51:27.351874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:51:27.351926 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:51:27.352013 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:51:27.352253 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.352759 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.353555 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:51:27.353760 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:51:27.353834 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:51:27.354089 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:51:27.354487 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:51:27.354576 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:51:27.354644 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 04:51:27.354693 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:51:27.360774 32446 raft_consensus.cc:1261] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 1 FOLLOWER]: Refusing update from remote peer 5e4fd13f45324405a35538ec3e42b51f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:27.360774 32246 raft_consensus.cc:1261] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 1 FOLLOWER]: Refusing update from remote peer 5e4fd13f45324405a35538ec3e42b51f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:51:27.361680 32548 consensus_queue.cc:1035] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0883807a964d0b8cc01f7008cc7124" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:51:27.361954 32557 consensus_queue.cc:1035] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "342c7ce1d9ea41388535ff9881351735" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:51:27.363379 32521 tablet.cc:2369] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:51:27.369837 32562 mvcc.cc:205] Tried to move back new op lower bound from 7316130559421947904 to 7316130559245336576. Current Snapshot: MvccSnapshot[applied={T|T < 7316130559421947904}]
I20260808 04:51:27.370543 32565 mvcc.cc:205] Tried to move back new op lower bound from 7316130559421947904 to 7316130559245336576. Current Snapshot: MvccSnapshot[applied={T|T < 7316130559421947904}]
I20260808 04:51:27.377736 32564 mvcc.cc:205] Tried to move back new op lower bound from 7316130559421947904 to 7316130559245336576. Current Snapshot: MvccSnapshot[applied={T|T < 7316130559421947904}]
I20260808 04:51:27.379689 31833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:27.379842 31833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:51:27.382388 31833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c72a326243144e7b8f868637dcd7d1ac: Sending DeleteTablet for 3 replicas of tablet 35124e3f3e944a93bde84f1faaeec976
I20260808 04:51:27.383319 32015 tablet_service.cc:1514] Processing DeleteTablet for tablet 35124e3f3e944a93bde84f1faaeec976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:27 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:49448
I20260808 04:51:27.383821 32580 tablet_replica.cc:330] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f: stopping tablet replica
I20260808 04:51:27.383809 32414 tablet_service.cc:1514] Processing DeleteTablet for tablet 35124e3f3e944a93bde84f1faaeec976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:27 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:37028
I20260808 04:51:27.384282 32580 raft_consensus.cc:2229] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:51:27.384353 32215 tablet_service.cc:1514] Processing DeleteTablet for tablet 35124e3f3e944a93bde84f1faaeec976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:51:27 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:41652
I20260808 04:51:27.384794 32580 raft_consensus.cc:2258] T 35124e3f3e944a93bde84f1faaeec976 P 5e4fd13f45324405a35538ec3e42b51f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:27.385380 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 31925
I20260808 04:51:27.386089 32582 tablet_replica.cc:330] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: stopping tablet replica
I20260808 04:51:27.386382 32582 raft_consensus.cc:2229] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:27.386584 32582 raft_consensus.cc:2258] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:27.394721 32581 tablet_replica.cc:330] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: stopping tablet replica
I20260808 04:51:27.395068 32581 raft_consensus.cc:2229] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:51:27.395411 32581 raft_consensus.cc:2258] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:51:27.395834 32582 ts_tablet_manager.cc:1905] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:27.396976 32581 ts_tablet_manager.cc:1905] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:51:27.398705 32582 ts_tablet_manager.cc:1918] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:27.398818 32582 log.cc:1196] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/35124e3f3e944a93bde84f1faaeec976
I20260808 04:51:27.399184 32582 ts_tablet_manager.cc:1939] T 35124e3f3e944a93bde84f1faaeec976 P 342c7ce1d9ea41388535ff9881351735: Deleting consensus metadata
I20260808 04:51:27.399909 32581 ts_tablet_manager.cc:1918] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:51:27.400012 32581 log.cc:1196] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/35124e3f3e944a93bde84f1faaeec976
I20260808 04:51:27.400385 32581 ts_tablet_manager.cc:1939] T 35124e3f3e944a93bde84f1faaeec976 P 7a0883807a964d0b8cc01f7008cc7124: Deleting consensus metadata
I20260808 04:51:27.400556 31817 catalog_manager.cc:4841] TS 342c7ce1d9ea41388535ff9881351735 (127.15.30.66:34161): tablet 35124e3f3e944a93bde84f1faaeec976 (table test-table [id=f25636fe55754b92a1a620acfd4cb297]) successfully deleted
W20260808 04:51:27.401338 31817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:46371 (error 108)
I20260808 04:51:27.401571 31816 catalog_manager.cc:4841] TS 7a0883807a964d0b8cc01f7008cc7124 (127.15.30.67:42419): tablet 35124e3f3e944a93bde84f1faaeec976 (table test-table [id=f25636fe55754b92a1a620acfd4cb297]) successfully deleted
I20260808 04:51:27.402267 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 32128
I20260808 04:51:27.402339 32534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:51:27.402062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46371 (local address 127.0.0.1:49470)
0808 04:51:27.402208 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 04:51:27.402269 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:51:27.409591 31817 catalog_manager.cc:4584] TS 5e4fd13f45324405a35538ec3e42b51f (127.15.30.65:46371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35124e3f3e944a93bde84f1faaeec976: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46371: connect: Connection refused (error 111)
I20260808 04:51:27.415743 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 32319
I20260808 04:51:27.429423 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 31790
2026-08-08T04:51:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2630 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:51:27 dist-test-slave-zpvx krb5kdc[32587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:51:27 dist-test-slave-zpvx krb5kdc[32587](info): set up 2 sockets
Aug 08 04:51:27 dist-test-slave-zpvx krb5kdc[32587](info): commencing operation
krb5kdc: starting...
W20260808 04:51:28.682698 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:51:28 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:51:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:51:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/rangeradmin_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/rangeradmin_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/rangerlookup_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/rangerlookup_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
I20260808 04:51:28.855568 15481 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdHhAno/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/postgres -l logfile start
2026-08-08 04:51:31.369 UTC [32629] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:51:31.369 UTC [32629] LOG: listening on IPv4 address "127.15.30.84", port 33059
2026-08-08 04:51:31.373 UTC [32629] LOG: listening on Unix socket "/tmp/.s.PGSQL.33059"
2026-08-08 04:51:31.395 UTC [32633] LOG: database system was shut down at 2026-08-08 04:51:29 UTC
2026-08-08 04:51:31.399 UTC [32629] LOG: database system is ready to accept connections
I20260808 04:51:32.570252 15481 mini_postgres.cc:96] Postgres bound to 33059
2026-08-08 04:51:32.578 UTC [32647] FATAL: database "slave" does not exist
127.15.30.84:33059 - accepting connections
I20260808 04:51:32.579607 15481 mini_ranger.cc:161] Starting Ranger...
I20260808 04:51:32.595310 15481 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:51:33.079352 15481 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:51:33.079479 15481 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-admin
2026-08-08 04:51:33,409 [I] DB FLAVOR :POSTGRES
2026-08-08 04:51:33,409 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:51:33,409 [I] Checking connection..
2026-08-08 04:51:33,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:51:33,683 [I] Checking connection passed.
2026-08-08 04:51:33,684 [I] --------- Verifying version history table ---------
2026-08-08 04:51:33,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:51:33,952 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:51:33,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:51:34,210 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:51:34,210 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:51:34,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:51:34.468 UTC [32732] WARNING: there is no transaction in progress
2026-08-08 04:51:34,477 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:51:34,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:51:34,732 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:51:34,732 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:51:34,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:51:34,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:51:35,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpvx.c.gcp-upstream.internal','N') ;"
2026-08-08 04:51:35,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:51:35,738 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:51:35,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:51:35,995 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:51:35,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:51:36,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:51:36,499 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:51:36,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:51:37.469 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.485 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.502 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.517 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.533 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.550 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.601 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.608 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.621 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.631 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.640 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.654 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.663 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.670 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.676 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:37.682 UTC [436] WARNING: there is no transaction in progress
2026-08-08 04:51:38,313 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:51:38,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:51:38,573 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:51:38,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:51:38,993 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:51:38,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:51:39,384 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:51:39,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:51:39,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:51:39,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpvx.c.gcp-upstream.internal';"
2026-08-08 04:51:40,249 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:51:40,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:51:40,571 [I] DB_PATCHES have already been applied
I20260808 04:51:40.577081 15481 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdHhAno/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:51:40.577147 15481 mini_ranger.cc:193] Using host: 127.15.30.84
I20260808 04:51:40.581766 15481 mini_ranger.cc:227] Ranger admin URL: http://127.15.30.84:44081
Aug 08, 2026 4:51:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:51:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:51:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:51:42.009699 15481 mini_ranger.cc:160] Time spent starting Ranger: real 9.430s user 0.000s sys 0.012s
Aug 08, 2026 4:52:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:52:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 4:52:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:52:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:52:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:52:04.370721 15481 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/rangerkms_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/rangerkms_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.30.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.30.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:52:04 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164724, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:52:04.470641 15481 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:52:04.482795 15481 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:52:05.316130 15481 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:52:05.316264 15481 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms
I20260808 04:52:05.316383 15481 mini_ranger_kms.cc:227] Using postgres at 127.15.30.84:33059
2026-08-08 04:52:06,318 [I] DB FLAVOR :POSTGRES
2026-08-08 04:52:06,318 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:52:06,318 [I] Checking connection
2026-08-08 04:52:06,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:52:06,617 [I] connection success
2026-08-08 04:52:06,617 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:52:06,618 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:52:06,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:52:06,897 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:52:06,898 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:52:06,898 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:52:06,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:33059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:52:07,180 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:52:07.694254 15481 mini_ranger_kms.cc:310] Created kms service
I20260808 04:52:07.912705 15481 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:52:07.956854 15481 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:52:07 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164724, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:08.390559 15481 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:52:08.390724 15481 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:52:08.390749 15481 mini_ranger_kms.cc:240] Using host: 127.15.30.84
I20260808 04:52:08.396595 15481 mini_ranger_kms.cc:276] Ranger KMS PID: 700
I20260808 04:52:08.396745 15481 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.30.84:54739
Aug 08, 2026 4:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:52:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:52:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:52:10 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164730, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.30.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:52:11.151461 15481 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.681s user 0.040s sys 0.664s
I20260808 04:52:11.151695 15481 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:52:11.151831 15481 mini_ranger_kms.cc:401] 127.15.30.84:54739/kms/v1/keys
Aug 08 04:52:11 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164730, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.30.84@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
Aug 08 04:52:11 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164730, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.30.84@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
Aug 08 04:52:14 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:52:14.279753 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33659
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33659
--ranger_config_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:54739/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:52:14.419981 770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:52:14.420223 770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:52:14.420264 770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:52:14.425031 770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:52:14.425146 770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:52:14.425215 770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:52:14.425256 770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:52:14.425307 770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:52:14.430923 770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.30.84:54739/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33659
--ranger_config_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:52:14.431941 770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:52:14.433008 770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:52:14.443128 778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:52:14.443229 779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:52:14.443117 781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:52:14.443459 770 server_base.cc:1034] running on GCE node
I20260808 04:52:14.444257 770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:52:14.445731 770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:52:14.446950 770 hybrid_clock.cc:648] HybridClock initialized: now 1786164734446913 us; error 52 us; skew 500 ppm
Aug 08 04:52:14 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164734, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:52:14.450639 770 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:52:14.451537 770 webserver.cc:460] Webserver started at http://127.15.30.126:41369/ using document root <none> and password file <none>
I20260808 04:52:14.452090 770 fs_manager.cc:359] Metadata directory not provided
I20260808 04:52:14.452220 770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:52:14.452505 770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:52:14 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164734, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:14.506820 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "641609dff94f493ab64a32c1a87924ab"
format_stamp: "Formatted at 2026-08-08 04:52:14 on dist-test-slave-zpvx"
server_key: "cd70ab1693e220a5c66a35271b208dde"
server_key_iv: "b1118ef5eaa305b66a000382c9decbbe"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:14.507557 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "641609dff94f493ab64a32c1a87924ab"
format_stamp: "Formatted at 2026-08-08 04:52:14 on dist-test-slave-zpvx"
server_key: "cd70ab1693e220a5c66a35271b208dde"
server_key_iv: "b1118ef5eaa305b66a000382c9decbbe"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:14.511868 770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:52:14.526223 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:14.527768 770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:52:14.527885 770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "641609dff94f493ab64a32c1a87924ab"
format_stamp: "Formatted at 2026-08-08 04:52:14 on dist-test-slave-zpvx"
server_key: "cd70ab1693e220a5c66a35271b208dde"
server_key_iv: "b1118ef5eaa305b66a000382c9decbbe"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:14.528039 770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:52:14.541023 770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:52:14.544147 770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:52:14.544355 770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:52:14.557590 770 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33659
I20260808 04:52:14.557730 861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33659 every 8 connection(s)
I20260808 04:52:14.558800 770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:52:14.561939 862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:52:14.568178 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 770
I20260808 04:52:14.568322 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:52:14.568358 862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab: Bootstrap starting.
I20260808 04:52:14.568665 15481 external_mini_cluster.cc:1428] Setting key e75a813cb9c80a8fec401f0d310aa7f4
I20260808 04:52:14.571575 862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab: Neither blocks nor log segments found. Creating new log.
I20260808 04:52:14.572657 862 log.cc:824] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:52:14.575302 862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab: No bootstrap required, opened a new log
Aug 08 04:52:14 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:52:14.581560 862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641609dff94f493ab64a32c1a87924ab" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33659 } }
I20260808 04:52:14.581950 862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:52:14.582036 862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641609dff94f493ab64a32c1a87924ab, State: Initialized, Role: FOLLOWER
I20260808 04:52:14.582525 862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641609dff94f493ab64a32c1a87924ab" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33659 } }
I20260808 04:52:14.582680 862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:52:14.582774 862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:52:14.582906 862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:52:14.584084 862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641609dff94f493ab64a32c1a87924ab" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33659 } }
I20260808 04:52:14.584517 862 leader_election.cc:302] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 641609dff94f493ab64a32c1a87924ab; no voters:
I20260808 04:52:14.584898 862 leader_election.cc:288] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:52:14.585343 871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:52:14.585677 871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [term 1 LEADER]: Becoming Leader. State: Replica: 641609dff94f493ab64a32c1a87924ab, State: Running, Role: LEADER
I20260808 04:52:14.586022 871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641609dff94f493ab64a32c1a87924ab" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33659 } }
I20260808 04:52:14.586275 867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:14.570425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.15.30.126:33659)
0808 04:52:14.570975 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:14.570995 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:14.571039 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:52:14.571952 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:14.571977 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:14.572447 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:14.572808 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:52:14.573810 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:14.575337 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:14.576209 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:14.576984 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:14.577142 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:14.580509 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:52:14.580516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:52:14.580533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:52:14.580565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:52:14.582867 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:14.583407 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:14.583413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:14.583419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:14.583546 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:14.583868 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:14.583871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:14.583873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:14.584402 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:52:14.584582 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:14.584962 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 04:52:14.585252 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":210,"threads_started":1}
I20260808 04:52:14.586563 862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:52:14.587565 875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 641609dff94f493ab64a32c1a87924ab. Latest consensus state: current_term: 1 leader_uuid: "641609dff94f493ab64a32c1a87924ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641609dff94f493ab64a32c1a87924ab" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33659 } } }
I20260808 04:52:14.587565 874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "641609dff94f493ab64a32c1a87924ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641609dff94f493ab64a32c1a87924ab" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33659 } } }
I20260808 04:52:14.587853 874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [sys.catalog]: This master's current role is: LEADER
I20260808 04:52:14.588078 875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab [sys.catalog]: This master's current role is: LEADER
I20260808 04:52:14.591578 883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:52:14.591989 862 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:52:14.595252 883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:52:14.609721 883 catalog_manager.cc:1347] Generated new cluster ID: 8c6042ca773a463e98835643f56905a4
I20260808 04:52:14.609803 883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:52:14.625211 883 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:52:14.626569 883 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:52:14.633378 883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab: Generated new TSK 0
I20260808 04:52:14.634337 883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:52:15.655436 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:54739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33659
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:52:15.825875 933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:52:15.826201 933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:52:15.826265 933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:52:15.832515 933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:52:15.832592 933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:52:15.832679 933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:52:15.840797 933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.30.84:54739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:52:15.842346 933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:52:15.843670 933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:52:15.856772 943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:52:15.856830 944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:52:15.857096 946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:52:15.858410 933 server_base.cc:1034] running on GCE node
I20260808 04:52:15.858952 933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:52:15.859792 933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:52:15.861065 933 hybrid_clock.cc:648] HybridClock initialized: now 1786164735861065 us; error 66 us; skew 500 ppm
Aug 08 04:52:15 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:52:15.865415 933 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:52:15.866497 933 webserver.cc:460] Webserver started at http://127.15.30.65:44027/ using document root <none> and password file <none>
I20260808 04:52:15.867211 933 fs_manager.cc:359] Metadata directory not provided
I20260808 04:52:15.867298 933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:52:15.867522 933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:52:15 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:15.884622 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "d62e4c3116f3438d9949b9fa65369c10"
format_stamp: "Formatted at 2026-08-08 04:52:15 on dist-test-slave-zpvx"
server_key: "75cdf70c52bc6d475361ce743ee62e79"
server_key_iv: "a8ae140e1d302804b384998d3f36adfc"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:15.885541 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "d62e4c3116f3438d9949b9fa65369c10"
format_stamp: "Formatted at 2026-08-08 04:52:15 on dist-test-slave-zpvx"
server_key: "75cdf70c52bc6d475361ce743ee62e79"
server_key_iv: "a8ae140e1d302804b384998d3f36adfc"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:15.890486 933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 04:52:15.908336 956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:15.910211 933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:52:15.910454 933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "d62e4c3116f3438d9949b9fa65369c10"
format_stamp: "Formatted at 2026-08-08 04:52:15 on dist-test-slave-zpvx"
server_key: "75cdf70c52bc6d475361ce743ee62e79"
server_key_iv: "a8ae140e1d302804b384998d3f36adfc"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:15.910699 933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:52:15.923761 933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:52:15.927939 933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:52:15.928222 933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:52:15.929199 933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:52:15.930399 933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:52:15.930503 933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:15.930594 933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:52:15.930673 933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:15.965189 933 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:43423
I20260808 04:52:15.966941 933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:52:15.967825 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 933
I20260808 04:52:15.967962 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:52:15.968482 15481 external_mini_cluster.cc:1428] Setting key 5fe7dd267896476d794be45e14cc0453
I20260808 04:52:15.967417 1120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:43423 every 8 connection(s)
Aug 08 04:52:15 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:52:15.982178 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:15.969213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:60705 (local address 127.15.30.126:33659)
0808 04:52:15.969701 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:15.969706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:15.970485 (+ 779us) server_negotiation.cc:407] Connection header received
0808 04:52:15.971683 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:15.971687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:15.971754 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:15.971871 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:52:15.973316 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:15.973885 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:15.975113 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:15.975349 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:15.975457 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:15.978385 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:52:15.978389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:52:15.978393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:52:15.978423 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:52:15.980023 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:15.980620 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:15.980624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:15.980626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:15.980713 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:15.981075 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:15.981078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:15.981080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:15.981263 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:52:15.981375 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:15.981866 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 04:52:15.981993 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":112,"threads_started":1}
I20260808 04:52:15.983068 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:15.969735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33659 (local address 127.15.30.65:60705)
0808 04:52:15.970327 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:15.970382 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:15.971455 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:15.972067 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:15.972074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:15.972524 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:52:15.973153 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:15.973169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:15.974018 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:15.974029 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:15.974963 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:15.974973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:15.975513 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:15.975517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:15.975742 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:15.976530 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:52:15.976554 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:52:15.978213 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:52:15.980163 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:52:15.980169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:52:15.980184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:52:15.980514 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:52:15.980818 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:52:15.980821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:52:15.980823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:52:15.980967 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:52:15.981374 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:52:15.981382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:52:15.981623 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 04:52:15.981806 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 04:52:15.982113 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":135,"threads_started":1}
I20260808 04:52:15.984356 1121 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33659
I20260808 04:52:15.984644 1121 heartbeater.cc:464] Registering TS with master...
I20260808 04:52:15.985152 1121 heartbeater.cc:511] Master 127.15.30.126:33659 requested a full tablet report, sending...
I20260808 04:52:15.986763 814 ts_manager.cc:194] Registered new tserver with Master: d62e4c3116f3438d9949b9fa65369c10 (127.15.30.65:43423)
I20260808 04:52:15.988806 814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:60705
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:52:16.021646 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:54739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33659
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:52:16.212991 1128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:52:16.213351 1128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:52:16.213433 1128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:52:16.217736 1128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:52:16.217789 1128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:52:16.217870 1128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:52:16.225639 1128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.30.84:54739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:52:16.226949 1128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:52:16.228276 1128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:52:16.243633 1128 server_base.cc:1034] running on GCE node
W20260808 04:52:16.243656 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:52:16.244176 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:52:16.244279 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:52:16.245270 1128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:52:16.246042 1128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:52:16.249367 1128 hybrid_clock.cc:648] HybridClock initialized: now 1786164736249336 us; error 45 us; skew 500 ppm
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:52:16.252871 1128 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:52:16.253859 1128 webserver.cc:460] Webserver started at http://127.15.30.66:36915/ using document root <none> and password file <none>
I20260808 04:52:16.254448 1128 fs_manager.cc:359] Metadata directory not provided
I20260808 04:52:16.254544 1128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:52:16.254774 1128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:16.270042 1128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "260351504db24ab29c34472e98e05293"
format_stamp: "Formatted at 2026-08-08 04:52:16 on dist-test-slave-zpvx"
server_key: "60c83b18365732b63ca9b3b45426e576"
server_key_iv: "a5b63cbc5cd8910619b5e9403cfcb6f5"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:16.270874 1128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "260351504db24ab29c34472e98e05293"
format_stamp: "Formatted at 2026-08-08 04:52:16 on dist-test-slave-zpvx"
server_key: "60c83b18365732b63ca9b3b45426e576"
server_key_iv: "a5b63cbc5cd8910619b5e9403cfcb6f5"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:16.275281 1128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:52:16.300117 1147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:16.301901 1128 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 04:52:16.302037 1128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "260351504db24ab29c34472e98e05293"
format_stamp: "Formatted at 2026-08-08 04:52:16 on dist-test-slave-zpvx"
server_key: "60c83b18365732b63ca9b3b45426e576"
server_key_iv: "a5b63cbc5cd8910619b5e9403cfcb6f5"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:16.302177 1128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:52:16.330681 1128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:52:16.335279 1128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:52:16.335575 1128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:52:16.336458 1128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:52:16.337807 1128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:52:16.337944 1128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:16.338112 1128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:52:16.338219 1128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:16.366837 1128 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:33003
I20260808 04:52:16.366921 1312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:33003 every 8 connection(s)
I20260808 04:52:16.368323 1128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:52:16.372942 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 1128
I20260808 04:52:16.373132 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:52:16.373476 15481 external_mini_cluster.cc:1428] Setting key 4ae211321c7d189c1683999e7e0ccf5c
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:52:16.414870 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.382366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:57151 (local address 127.15.30.126:33659)
0808 04:52:16.382525 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:16.382530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:16.389106 (+ 6576us) server_negotiation.cc:407] Connection header received
0808 04:52:16.391172 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:16.391177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:16.391253 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:16.391314 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:52:16.393450 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.394417 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.396188 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.396594 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.396678 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.410334 (+ 13656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:52:16.410341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:52:16.410343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:52:16.410381 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:52:16.411944 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:16.412630 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:16.412637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:16.412641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:16.412734 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:16.413187 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:16.413193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:16.413198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:16.413472 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:52:16.413579 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:16.414402 (+ 823us) server_negotiation.cc:299] Negotiation successful
0808 04:52:16.414583 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:52:16.416540 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.380913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33659 (local address 127.15.30.66:57151)
0808 04:52:16.388889 (+ 7976us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:16.388962 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:16.390508 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:16.391758 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:16.391768 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:16.392410 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:52:16.393262 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.393286 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.394597 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.394600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.396026 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.396038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.396905 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.396909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.397195 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.407799 (+ 10604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:52:16.407828 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:52:16.410054 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:52:16.412107 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:52:16.412114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:52:16.412132 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:52:16.412490 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:52:16.412853 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:52:16.412859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:52:16.412864 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:52:16.413064 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:52:16.413657 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:52:16.413667 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:52:16.414058 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 04:52:16.415072 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0808 04:52:16.415464 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7668,"thread_start_us":310,"threads_started":1}
I20260808 04:52:16.418083 1313 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33659
I20260808 04:52:16.418349 1313 heartbeater.cc:464] Registering TS with master...
I20260808 04:52:16.418882 1313 heartbeater.cc:511] Master 127.15.30.126:33659 requested a full tablet report, sending...
I20260808 04:52:16.420120 814 ts_manager.cc:194] Registered new tserver with Master: 260351504db24ab29c34472e98e05293 (127.15.30.66:33003)
I20260808 04:52:16.420958 814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:57151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:52:16.432610 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:54739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33659
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:52:16.622929 1318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:52:16.623404 1318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:52:16.623534 1318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:52:16.629848 1318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:52:16.630007 1318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:52:16.630177 1318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:52:16.639230 1318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.30.84:54739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:52:16.640810 1318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:52:16.642278 1318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:52:16.658056 1335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:52:16.658618 1330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:52:16.659804 1318 server_base.cc:1034] running on GCE node
W20260808 04:52:16.660717 1338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:52:16.661372 1318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:52:16.662242 1318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:52:16.663537 1318 hybrid_clock.cc:648] HybridClock initialized: now 1786164736663450 us; error 105 us; skew 500 ppm
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:52:16.667951 1318 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:52:16.669266 1318 webserver.cc:460] Webserver started at http://127.15.30.67:39467/ using document root <none> and password file <none>
I20260808 04:52:16.670058 1318 fs_manager.cc:359] Metadata directory not provided
I20260808 04:52:16.670233 1318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:52:16.670557 1318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:16.698566 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b"
format_stamp: "Formatted at 2026-08-08 04:52:16 on dist-test-slave-zpvx"
server_key: "37a51beb2a766b4c828147a311a8b8f1"
server_key_iv: "33fcc3caa309f4375e4a339c34a7eb83"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:16.700700 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b"
format_stamp: "Formatted at 2026-08-08 04:52:16 on dist-test-slave-zpvx"
server_key: "37a51beb2a766b4c828147a311a8b8f1"
server_key_iv: "33fcc3caa309f4375e4a339c34a7eb83"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:16.709074 1318 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:16.741194 1347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:16.743410 1318 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260808 04:52:16.743610 1318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b"
format_stamp: "Formatted at 2026-08-08 04:52:16 on dist-test-slave-zpvx"
server_key: "37a51beb2a766b4c828147a311a8b8f1"
server_key_iv: "33fcc3caa309f4375e4a339c34a7eb83"
server_key_version: "kuduclusterkey@0"
I20260808 04:52:16.743809 1318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:52:16.761251 1318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:52:16.764845 1318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:52:16.765084 1318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:52:16.765796 1318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:52:16.766886 1318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:52:16.766983 1318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:16.767068 1318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:52:16.767122 1318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:52:16.782548 1318 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:35657
I20260808 04:52:16.783833 1318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:52:16.790705 1526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:35657 every 8 connection(s)
I20260808 04:52:16.790874 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 1318
I20260808 04:52:16.791004 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:52:16.791347 15481 external_mini_cluster.cc:1428] Setting key 1d8f31c1005c4166a8ab6d893b8292db
Aug 08 04:52:16 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:52:16.834497 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.796178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:50569 (local address 127.15.30.126:33659)
0808 04:52:16.796296 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:16.796301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:16.796785 (+ 484us) server_negotiation.cc:407] Connection header received
0808 04:52:16.798406 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:16.798410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:16.798465 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:16.798515 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:52:16.800869 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.801703 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.817817 (+ 16114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.818176 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.818257 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.825255 (+ 6998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:52:16.825259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:52:16.825262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:52:16.825293 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:52:16.827005 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:16.831412 (+ 4407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:16.831416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:16.831418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:16.831515 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:16.831980 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:16.831983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:16.831985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:16.832221 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:52:16.832310 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:16.834201 (+ 1891us) server_negotiation.cc:299] Negotiation successful
0808 04:52:16.834331 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:52:16.836186 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.795034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33659 (local address 127.15.30.67:50569)
0808 04:52:16.796638 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:16.796694 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:16.798032 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:16.798928 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:16.798937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:16.799507 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:52:16.800696 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.800719 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.814874 (+ 14155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.814877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.817660 (+ 2783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.817674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.818480 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.818483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.818742 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.821883 (+ 3141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:52:16.821907 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:52:16.825041 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:52:16.827159 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:52:16.827165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:52:16.827183 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:52:16.827517 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:52:16.831631 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:52:16.831634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:52:16.831636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:52:16.831866 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:52:16.832910 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:52:16.832916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:52:16.833884 (+ 968us) client_negotiation.cc:769] Sending connection context
0808 04:52:16.834720 (+ 836us) client_negotiation.cc:240] Negotiation successful
0808 04:52:16.835066 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":166,"threads_started":1}
I20260808 04:52:16.837462 1528 heartbeater.cc:347] Connected to a master server at 127.15.30.126:33659
I20260808 04:52:16.837724 1528 heartbeater.cc:464] Registering TS with master...
I20260808 04:52:16.838277 1528 heartbeater.cc:511] Master 127.15.30.126:33659 requested a full tablet report, sending...
I20260808 04:52:16.839530 814 ts_manager.cc:194] Registered new tserver with Master: 4576f8b7b3eb4ac8ad376575e3c1a02b (127.15.30.67:35657)
I20260808 04:52:16.840247 814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:50569
I20260808 04:52:16.845249 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:52:16.859179 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.847686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.15.30.126:33659)
0808 04:52:16.847803 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:16.847807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:16.848837 (+ 1030us) server_negotiation.cc:407] Connection header received
0808 04:52:16.849053 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:16.849057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:16.849106 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:16.849162 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:52:16.850340 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.851233 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.852684 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.853072 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.853152 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.855193 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:52:16.855197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:52:16.855201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:52:16.855228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:52:16.857175 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:16.857721 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:16.857725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:16.857727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:16.857818 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:52:16.858153 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:52:16.858156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:52:16.858159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:52:16.858376 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:52:16.858484 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:16.858871 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 04:52:16.858995 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:52:16.863159 814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:52:16.867905 814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:52:16.891530 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.878515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:43423 (local address 127.0.0.1:52744)
0808 04:52:16.883527 (+ 5012us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:16.883586 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:16.883717 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:16.884537 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:16.884543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:16.884568 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:52:16.884746 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.884756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.886950 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.886954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.888515 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.888530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.890941 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.890945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.891072 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.891114 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:52:16.891208 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:52:16.891326 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3631,"thread_start_us":136,"threads_started":1}
I20260808 04:52:16.892299 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.879764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.15.30.65:43423)
0808 04:52:16.882072 (+ 2308us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:16.882080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:16.883903 (+ 1823us) server_negotiation.cc:407] Connection header received
0808 04:52:16.883987 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:16.883992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:16.884197 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:16.884336 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:52:16.884874 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.886811 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.888773 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.889699 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.890690 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.890879 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:16.891947 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0808 04:52:16.892093 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2196,"thread_start_us":99,"threads_started":1}
I20260808 04:52:16.896044 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 22f7be64dc734a2bb8c124359e985ac9 (DEFAULT_TABLE table=test-table [id=44e215320fb943e6a5b49ba77d031ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:52:16.897428 1011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f7be64dc734a2bb8c124359e985ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:52:16.899668 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.880793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:35657 (local address 127.0.0.1:35310)
0808 04:52:16.884552 (+ 3759us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:16.884566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:16.884685 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:16.889539 (+ 4854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:16.889542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:16.889561 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:52:16.889664 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.889671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.894185 (+ 4514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.894189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.895294 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.895306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.899257 (+ 3951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.899261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.899348 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.899370 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:52:16.899442 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:52:16.899520 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3696,"thread_start_us":84,"threads_started":1}
I20260808 04:52:16.900518 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.881076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:33003 (local address 127.0.0.1:37068)
0808 04:52:16.890757 (+ 9681us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:16.890771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:16.890931 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:16.891671 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:16.891675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:16.891699 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:52:16.891821 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.891827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.895426 (+ 3599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.895429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.896557 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:16.896568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.899977 (+ 3409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.899981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:16.900190 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.900217 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:52:16.900302 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:52:16.900392 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9618,"thread_start_us":100,"threads_started":1}
I20260808 04:52:16.908782 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.881491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.15.30.66:33003)
0808 04:52:16.888748 (+ 7257us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:16.888813 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:16.891028 (+ 2215us) server_negotiation.cc:407] Connection header received
0808 04:52:16.891122 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:16.891127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:16.891316 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:16.891504 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:52:16.891956 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.893235 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.896813 (+ 3578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.897728 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.897831 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.897967 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:16.908450 (+ 10483us) server_negotiation.cc:299] Negotiation successful
0808 04:52:16.908566 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7116,"thread_start_us":124,"threads_started":1}
I20260808 04:52:16.909430 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:16.881239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.15.30.67:35657)
0808 04:52:16.888999 (+ 7760us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:16.889010 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:16.889049 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:52:16.889130 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:16.889135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:16.889315 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:16.889468 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:52:16.889946 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.894038 (+ 4092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.898051 (+ 4013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:16.898904 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:16.899027 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:16.899187 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:16.909087 (+ 9900us) server_negotiation.cc:299] Negotiation successful
0808 04:52:16.909217 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7663,"thread_start_us":149,"threads_started":1}
I20260808 04:52:16.913582 1210 tablet_service.cc:1467] Processing CreateTablet for tablet 22f7be64dc734a2bb8c124359e985ac9 (DEFAULT_TABLE table=test-table [id=44e215320fb943e6a5b49ba77d031ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:52:16.914278 1553 tablet_bootstrap.cc:492] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10: Bootstrap starting.
I20260808 04:52:16.916515 1402 tablet_service.cc:1467] Processing CreateTablet for tablet 22f7be64dc734a2bb8c124359e985ac9 (DEFAULT_TABLE table=test-table [id=44e215320fb943e6a5b49ba77d031ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:52:16.917040 1553 tablet_bootstrap.cc:654] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10: Neither blocks nor log segments found. Creating new log.
I20260808 04:52:16.917620 1402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f7be64dc734a2bb8c124359e985ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:52:16.918432 1553 log.cc:824] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10: Log is configured to *not* fsync() on all Append() calls
I20260808 04:52:16.919785 1210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f7be64dc734a2bb8c124359e985ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:52:16.927583 1557 tablet_bootstrap.cc:492] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Bootstrap starting.
I20260808 04:52:16.931443 1553 tablet_bootstrap.cc:492] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10: No bootstrap required, opened a new log
I20260808 04:52:16.931452 1557 tablet_bootstrap.cc:654] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Neither blocks nor log segments found. Creating new log.
I20260808 04:52:16.931695 1553 ts_tablet_manager.cc:1397] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260808 04:52:16.932840 1557 log.cc:824] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:52:16.933990 1558 tablet_bootstrap.cc:492] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Bootstrap starting.
I20260808 04:52:16.936805 1558 tablet_bootstrap.cc:654] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Neither blocks nor log segments found. Creating new log.
I20260808 04:52:16.938071 1558 log.cc:824] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Log is configured to *not* fsync() on all Append() calls
I20260808 04:52:16.938838 1553 raft_consensus.cc:348] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:16.939110 1553 raft_consensus.cc:374] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:52:16.939184 1553 raft_consensus.cc:729] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62e4c3116f3438d9949b9fa65369c10, State: Initialized, Role: FOLLOWER
I20260808 04:52:16.939692 1553 consensus_queue.cc:260] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:16.940245 1557 tablet_bootstrap.cc:492] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: No bootstrap required, opened a new log
I20260808 04:52:16.940498 1557 ts_tablet_manager.cc:1397] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260808 04:52:16.941006 1553 ts_tablet_manager.cc:1428] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260808 04:52:16.941408 1121 heartbeater.cc:503] Master 127.15.30.126:33659 was elected leader, sending a full tablet report...
I20260808 04:52:16.946444 1558 tablet_bootstrap.cc:492] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: No bootstrap required, opened a new log
I20260808 04:52:16.946278 1557 raft_consensus.cc:348] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:16.946636 1558 ts_tablet_manager.cc:1397] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260808 04:52:16.946632 1557 raft_consensus.cc:374] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:52:16.946919 1557 raft_consensus.cc:729] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4576f8b7b3eb4ac8ad376575e3c1a02b, State: Initialized, Role: FOLLOWER
I20260808 04:52:16.947701 1557 consensus_queue.cc:260] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:16.948807 1557 ts_tablet_manager.cc:1428] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260808 04:52:16.949258 1528 heartbeater.cc:503] Master 127.15.30.126:33659 was elected leader, sending a full tablet report...
I20260808 04:52:16.953310 1558 raft_consensus.cc:348] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:16.953615 1558 raft_consensus.cc:374] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:52:16.953696 1558 raft_consensus.cc:729] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 260351504db24ab29c34472e98e05293, State: Initialized, Role: FOLLOWER
I20260808 04:52:16.954165 1558 consensus_queue.cc:260] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:16.955247 1558 ts_tablet_manager.cc:1428] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260808 04:52:16.955649 1313 heartbeater.cc:503] Master 127.15.30.126:33659 was elected leader, sending a full tablet report...
W20260808 04:52:16.975153 1122 tablet.cc:2369] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:52:17.020354 1560 raft_consensus.cc:482] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:52:17.020617 1560 raft_consensus.cc:504] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:17.022038 1560 leader_election.cc:288] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4576f8b7b3eb4ac8ad376575e3c1a02b (127.15.30.67:35657), 260351504db24ab29c34472e98e05293 (127.15.30.66:33003)
I20260808 04:52:17.034904 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:17.023947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:54545 (local address 127.15.30.66:33003)
0808 04:52:17.024062 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:17.024066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:17.024764 (+ 698us) server_negotiation.cc:407] Connection header received
0808 04:52:17.025448 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:17.025451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:17.025503 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:17.025552 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:52:17.027216 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.027968 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.032262 (+ 4294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.033010 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.033110 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:17.033173 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:17.034692 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0808 04:52:17.034772 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:52:17.035621 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:17.023446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:33003 (local address 127.15.30.65:54545)
0808 04:52:17.024638 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:17.024652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:17.025037 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:17.025953 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:17.025956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:17.025971 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:52:17.026109 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:17.026115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.030288 (+ 4173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.030301 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:17.032103 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:17.032115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.033391 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.033396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:17.033480 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:17.034422 (+ 942us) client_negotiation.cc:769] Sending connection context
0808 04:52:17.035316 (+ 894us) client_negotiation.cc:240] Negotiation successful
0808 04:52:17.035401 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":158,"threads_started":1}
I20260808 04:52:17.036206 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:17.023080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:51507 (local address 127.15.30.67:35657)
0808 04:52:17.023228 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:17.023233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:17.024358 (+ 1125us) server_negotiation.cc:407] Connection header received
0808 04:52:17.025177 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:17.025181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:17.025234 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:17.025286 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:52:17.026253 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.027034 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.033594 (+ 6560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.034144 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.034242 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:17.034286 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:17.036000 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0808 04:52:17.036082 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:52:17.037696 1245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f7be64dc734a2bb8c124359e985ac9" candidate_uuid: "d62e4c3116f3438d9949b9fa65369c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "260351504db24ab29c34472e98e05293" is_pre_election: true
I20260808 04:52:17.038048 1245 raft_consensus.cc:2454] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62e4c3116f3438d9949b9fa65369c10 in term 0.
I20260808 04:52:17.038741 962 leader_election.cc:302] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 260351504db24ab29c34472e98e05293, d62e4c3116f3438d9949b9fa65369c10; no voters:
I20260808 04:52:17.039067 1560 raft_consensus.cc:2781] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:52:17.039140 1560 raft_consensus.cc:482] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:52:17.039196 1560 raft_consensus.cc:3037] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:52:17.040904 1560 raft_consensus.cc:504] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:17.041291 1560 leader_election.cc:288] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [CANDIDATE]: Term 1 election: Requested vote from peers 4576f8b7b3eb4ac8ad376575e3c1a02b (127.15.30.67:35657), 260351504db24ab29c34472e98e05293 (127.15.30.66:33003)
W20260808 04:52:17.041543 1529 tablet.cc:2369] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:52:17.041873 1245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f7be64dc734a2bb8c124359e985ac9" candidate_uuid: "d62e4c3116f3438d9949b9fa65369c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "260351504db24ab29c34472e98e05293"
I20260808 04:52:17.041993 1245 raft_consensus.cc:3037] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:52:17.043956 1245 raft_consensus.cc:2454] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62e4c3116f3438d9949b9fa65369c10 in term 1.
I20260808 04:52:17.036931 1565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:17.022610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:35657 (local address 127.15.30.65:51507)
0808 04:52:17.024222 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:17.024236 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:52:17.024917 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:52:17.025694 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:52:17.025698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:52:17.025715 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:52:17.025819 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:17.025825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.028123 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.028127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:17.030016 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:52:17.030038 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.034378 (+ 4340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.034382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:52:17.034571 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:17.034588 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 04:52:17.036685 (+ 2097us) client_negotiation.cc:240] Negotiation successful
0808 04:52:17.036759 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"mutex_wait_us":100,"thread_start_us":125,"threads_started":1}
I20260808 04:52:17.044301 962 leader_election.cc:302] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 260351504db24ab29c34472e98e05293, d62e4c3116f3438d9949b9fa65369c10; no voters:
I20260808 04:52:17.044627 1560 raft_consensus.cc:2781] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:52:17.044869 1560 raft_consensus.cc:686] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [term 1 LEADER]: Becoming Leader. State: Replica: d62e4c3116f3438d9949b9fa65369c10, State: Running, Role: LEADER
I20260808 04:52:17.045128 1451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f7be64dc734a2bb8c124359e985ac9" candidate_uuid: "d62e4c3116f3438d9949b9fa65369c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" is_pre_election: true
I20260808 04:52:17.045274 1560 consensus_queue.cc:237] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } }
I20260808 04:52:17.045424 1451 raft_consensus.cc:2454] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62e4c3116f3438d9949b9fa65369c10 in term 0.
I20260808 04:52:17.045789 1449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f7be64dc734a2bb8c124359e985ac9" candidate_uuid: "d62e4c3116f3438d9949b9fa65369c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b"
I20260808 04:52:17.045892 1449 raft_consensus.cc:3037] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:52:17.047502 1449 raft_consensus.cc:2454] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62e4c3116f3438d9949b9fa65369c10 in term 1.
I20260808 04:52:17.049145 814 catalog_manager.cc:5495] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 reported cstate change: term changed from 0 to 1, leader changed from <none> to d62e4c3116f3438d9949b9fa65369c10 (127.15.30.65). New cstate: current_term: 1 leader_uuid: "d62e4c3116f3438d9949b9fa65369c10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d62e4c3116f3438d9949b9fa65369c10" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 } health_report { overall_health: UNKNOWN } } }
I20260808 04:52:17.097213 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:17.092304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.15.30.65:43423)
0808 04:52:17.092475 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:52:17.092480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:52:17.092495 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:52:17.092582 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:52:17.092585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:52:17.092664 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:52:17.092779 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:52:17.093279 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.094038 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.095298 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:52:17.095640 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:52:17.095745 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:52:17.096095 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:52:17.096688 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:52:17.096804 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:52:17.096916 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 04:52:17.096975 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 04:52:17.105301 1449 raft_consensus.cc:1261] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 1 FOLLOWER]: Refusing update from remote peer d62e4c3116f3438d9949b9fa65369c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:52:17.105312 1245 raft_consensus.cc:1261] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 1 FOLLOWER]: Refusing update from remote peer d62e4c3116f3438d9949b9fa65369c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:52:17.106580 1568 consensus_queue.cc:1035] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4576f8b7b3eb4ac8ad376575e3c1a02b" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:52:17.107082 1560 consensus_queue.cc:1035] T 22f7be64dc734a2bb8c124359e985ac9 P d62e4c3116f3438d9949b9fa65369c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "260351504db24ab29c34472e98e05293" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 33003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:52:17.120821 1314 tablet.cc:2369] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:52:17.126886 1576 mvcc.cc:205] Tried to move back new op lower bound from 7316130763173093376 to 7316130762942099456. Current Snapshot: MvccSnapshot[applied={T|T < 7316130763173093376}]
I20260808 04:52:17.129544 1574 mvcc.cc:205] Tried to move back new op lower bound from 7316130763173093376 to 7316130762942099456. Current Snapshot: MvccSnapshot[applied={T|T < 7316130763173093376}]
I20260808 04:52:17.139056 814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:52:17.139247 814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:52:17.142182 814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 641609dff94f493ab64a32c1a87924ab: Sending DeleteTablet for 3 replicas of tablet 22f7be64dc734a2bb8c124359e985ac9
I20260808 04:52:17.143448 1402 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f7be64dc734a2bb8c124359e985ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:52:17 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:35310
I20260808 04:52:17.143932 1210 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f7be64dc734a2bb8c124359e985ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:52:17 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:37068
I20260808 04:52:17.144852 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f7be64dc734a2bb8c124359e985ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:52:17 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:52744
I20260808 04:52:17.146138 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 933
I20260808 04:52:17.151760 1586 tablet_replica.cc:330] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: stopping tablet replica
I20260808 04:52:17.152128 1586 raft_consensus.cc:2229] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:52:17.152524 1586 raft_consensus.cc:2258] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:52:17.154556 1586 ts_tablet_manager.cc:1905] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:52:17.155296 1587 tablet_replica.cc:330] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: stopping tablet replica
I20260808 04:52:17.155647 1587 raft_consensus.cc:2229] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:52:17.156041 1587 raft_consensus.cc:2258] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:52:17.159732 1586 ts_tablet_manager.cc:1918] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:52:17.159885 1586 log.cc:1196] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/22f7be64dc734a2bb8c124359e985ac9
I20260808 04:52:17.160902 1586 ts_tablet_manager.cc:1939] T 22f7be64dc734a2bb8c124359e985ac9 P 4576f8b7b3eb4ac8ad376575e3c1a02b: Deleting consensus metadata
I20260808 04:52:17.163272 792 catalog_manager.cc:4841] TS 4576f8b7b3eb4ac8ad376575e3c1a02b (127.15.30.67:35657): tablet 22f7be64dc734a2bb8c124359e985ac9 (table test-table [id=44e215320fb943e6a5b49ba77d031ab5]) successfully deleted
Aug 08 04:52:17 dist-test-slave-zpvx krb5kdc[32587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164736, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:17.165493 1587 ts_tablet_manager.cc:1905] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 04:52:17.168421 791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:43423 (error 108)
I20260808 04:52:17.168882 1587 ts_tablet_manager.cc:1918] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:52:17.168977 1587 log.cc:1196] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/22f7be64dc734a2bb8c124359e985ac9
I20260808 04:52:17.169397 1587 ts_tablet_manager.cc:1939] T 22f7be64dc734a2bb8c124359e985ac9 P 260351504db24ab29c34472e98e05293: Deleting consensus metadata
I20260808 04:52:17.170213 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 1128
I20260808 04:52:17.170279 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:17.169937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:43423 (local address 127.0.0.1:52758)
0808 04:52:17.170100 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:17.170182 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:43423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:52:17.178862 791 catalog_manager.cc:4584] TS d62e4c3116f3438d9949b9fa65369c10 (127.15.30.65:43423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22f7be64dc734a2bb8c124359e985ac9: Network error: Client connection negotiation failed: client connection to 127.15.30.65:43423: connect: Connection refused (error 111)
I20260808 04:52:17.186650 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 1318
I20260808 04:52:17.200249 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:52:17.199874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:33003 (local address 127.0.0.1:37078)
0808 04:52:17.200036 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:52:17.200135 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.66:33003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:52:17.213642 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 770
I20260808 04:52:17.232144 15481 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:52:17.578295 15481 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:52:17.578466 15481 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:52:17.933352 15481 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:52:17.933 UTC [32629] LOG: received smart shutdown request
2026-08-08 04:52:17.937 UTC [32629] LOG: background worker "logical replication launcher" (PID 32640) exited with exit code 1
2026-08-08 04:52:17.937 UTC [32634] LOG: shutting down
2026-08-08 04:52:18.105 UTC [32629] LOG: database system is shut down
2026-08-08T04:52:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50809 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:52:18 dist-test-slave-zpvx krb5kdc[1605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:52:18 dist-test-slave-zpvx krb5kdc[1605](info): set up 2 sockets
Aug 08 04:52:18 dist-test-slave-zpvx krb5kdc[1605](info): commencing operation
krb5kdc: starting...
W20260808 04:52:19.505172 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:52:19 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:52:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/rangeradmin_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/rangeradmin_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/rangerlookup_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/rangerlookup_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
I20260808 04:52:19.674868 15481 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdHhAno/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/postgres -l logfile start
2026-08-08 04:52:22.135 UTC [1651] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:52:22.135 UTC [1651] LOG: listening on IPv4 address "127.15.30.84", port 36833
2026-08-08 04:52:22.139 UTC [1651] LOG: listening on Unix socket "/tmp/.s.PGSQL.36833"
2026-08-08 04:52:22.161 UTC [1659] LOG: database system was shut down at 2026-08-08 04:52:20 UTC
2026-08-08 04:52:22.165 UTC [1651] LOG: database system is ready to accept connections
I20260808 04:52:23.366648 15481 mini_postgres.cc:96] Postgres bound to 36833
2026-08-08 04:52:23.374 UTC [1671] FATAL: database "slave" does not exist
127.15.30.84:36833 - accepting connections
I20260808 04:52:23.374853 15481 mini_ranger.cc:161] Starting Ranger...
I20260808 04:52:23.390085 15481 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:52:23.828269 15481 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:52:23.828414 15481 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-admin
2026-08-08 04:52:24,106 [I] DB FLAVOR :POSTGRES
2026-08-08 04:52:24,107 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:52:24,107 [I] Checking connection..
2026-08-08 04:52:24,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:52:24,370 [I] Checking connection passed.
2026-08-08 04:52:24,370 [I] --------- Verifying version history table ---------
2026-08-08 04:52:24,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:52:24,598 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:52:24,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:52:24,846 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:52:24,846 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:52:24,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:52:25.086 UTC [1757] WARNING: there is no transaction in progress
2026-08-08 04:52:25,092 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:52:25,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:52:25,355 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:52:25,355 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:52:25,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:52:25,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:52:25,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpvx.c.gcp-upstream.internal','N') ;"
2026-08-08 04:52:26,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:52:26,336 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:52:26,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:52:26,611 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:52:26,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:52:26,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:52:27,122 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:52:27,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:52:28.030 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.045 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.061 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.075 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.090 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.110 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.173 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.180 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.193 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.202 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.212 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.224 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.233 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.239 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.246 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28.252 UTC [1916] WARNING: there is no transaction in progress
2026-08-08 04:52:28,846 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:52:28,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:52:29,102 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:52:29,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:52:29,366 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:52:29,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:52:29,605 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:52:29,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:52:29,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:52:30,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpvx.c.gcp-upstream.internal';"
2026-08-08 04:52:30,294 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:52:30,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:52:30,554 [I] DB_PATCHES have already been applied
I20260808 04:52:30.560429 15481 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdHhAno/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:52:30.560498 15481 mini_ranger.cc:193] Using host: 127.15.30.84
I20260808 04:52:30.565080 15481 mini_ranger.cc:227] Ranger admin URL: http://127.15.30.84:34375
Aug 08, 2026 4:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:52:31.968007 15481 mini_ranger.cc:160] Time spent starting Ranger: real 8.593s user 0.010s sys 0.002s
Aug 08, 2026 4:52:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:52:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 4:52:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:52:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:52:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:52:53.799175 15481 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.30.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.30.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/rangerkms_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.30.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/rangerkms_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.30.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.30.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.30.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/HTTP_127.15.30.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:52:53 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164773, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:52:53.904347 15481 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:52:53.915396 15481 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:52:54.790647 15481 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:52:54.790742 15481 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms
I20260808 04:52:54.790766 15481 mini_ranger_kms.cc:227] Using postgres at 127.15.30.84:36833
2026-08-08 04:52:54,991 [I] DB FLAVOR :POSTGRES
2026-08-08 04:52:54,991 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:52:54,991 [I] Checking connection
2026-08-08 04:52:54,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:52:55,229 [I] connection success
2026-08-08 04:52:55,229 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:52:55,229 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:52:55,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:52:55,475 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:52:55,476 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:52:55,476 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:52:55,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.30.84:36833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:52:55,763 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:52:56.216751 15481 mini_ranger_kms.cc:310] Created kms service
I20260808 04:52:56.386385 15481 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:52:56.432281 15481 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:52:56 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164773, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:52:56.851161 15481 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:52:56.851325 15481 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdHhAno/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdHhAno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:52:56.851348 15481 mini_ranger_kms.cc:240] Using host: 127.15.30.84
I20260808 04:52:56.856892 15481 mini_ranger_kms.cc:276] Ranger KMS PID: 2183
I20260808 04:52:56.856978 15481 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.30.84:56331
Aug 08, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:52:59 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164779, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.30.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:52:59.580814 15481 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.676s user 0.000s sys 0.182s
I20260808 04:52:59.580963 15481 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:52:59.581037 15481 mini_ranger_kms.cc:401] 127.15.30.84:56331/kms/v1/keys
Aug 08 04:52:59 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164779, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.30.84@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
Aug 08 04:53:00 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164779, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.30.84@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
Aug 08 04:53:02 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:53:02.511292 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42569
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:42569
--ranger_config_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:02.652195 2253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:02.652518 2253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:02.652598 2253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:02.656738 2253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:02.656797 2253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:02.656816 2253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:02.656832 2253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:02.656847 2253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:02.662708 2253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.30.84:56331/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:42569
--ranger_config_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:02.663800 2253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:02.664891 2253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:02.674716 2261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:02.674746 2263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:02.674841 2253 server_base.cc:1034] running on GCE node
W20260808 04:53:02.674770 2259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:02.675520 2253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:02.676769 2253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:02.677963 2253 hybrid_clock.cc:648] HybridClock initialized: now 1786164782677944 us; error 32 us; skew 500 ppm
Aug 08 04:53:02 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164782, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:02.681001 2253 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:02.681857 2253 webserver.cc:460] Webserver started at http://127.15.30.126:38097/ using document root <none> and password file <none>
I20260808 04:53:02.682426 2253 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:02.682511 2253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:02.682734 2253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:53:02 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164782, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:02.743903 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "6d1959434b934db4982aca1a59fda0f4"
format_stamp: "Formatted at 2026-08-08 04:53:02 on dist-test-slave-zpvx"
server_key: "586caf0e518ba0680f94a719594c0fc5"
server_key_iv: "44267af0e0bf653c6387cc438a29f486"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:02.744572 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "6d1959434b934db4982aca1a59fda0f4"
format_stamp: "Formatted at 2026-08-08 04:53:02 on dist-test-slave-zpvx"
server_key: "586caf0e518ba0680f94a719594c0fc5"
server_key_iv: "44267af0e0bf653c6387cc438a29f486"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:02.748623 2253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:53:02.762605 2270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:02.763980 2253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:53:02.764140 2253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "6d1959434b934db4982aca1a59fda0f4"
format_stamp: "Formatted at 2026-08-08 04:53:02 on dist-test-slave-zpvx"
server_key: "586caf0e518ba0680f94a719594c0fc5"
server_key_iv: "44267af0e0bf653c6387cc438a29f486"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:02.764276 2253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:02.775074 2253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:02.777818 2253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:02.778012 2253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:02.789516 2253 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:42569
I20260808 04:53:02.789592 2353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:42569 every 8 connection(s)
I20260808 04:53:02.790577 2253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:02.793479 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:02.799006 2355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Bootstrap starting.
I20260808 04:53:02.799744 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 2253
I20260808 04:53:02.799886 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:53:02.800164 15481 external_mini_cluster.cc:1428] Setting key 724685247ba18a4225be8d33736625ef
I20260808 04:53:02.801981 2355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:02.802754 2355 log.cc:824] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:02.805020 2355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: No bootstrap required, opened a new log
I20260808 04:53:02.810371 2355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:02.810618 2355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 08 04:53:02 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:02.810711 2355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1959434b934db4982aca1a59fda0f4, State: Initialized, Role: FOLLOWER
I20260808 04:53:02.811259 2355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:02.811412 2355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:02.811503 2355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:02.811606 2355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:02.812680 2355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:02.813032 2355 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1959434b934db4982aca1a59fda0f4; no voters:
I20260808 04:53:02.813356 2355 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:53:02.813593 2362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:02.814004 2362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 6d1959434b934db4982aca1a59fda0f4, State: Running, Role: LEADER
I20260808 04:53:02.814404 2362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:02.814798 2355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:02.815773 2362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1959434b934db4982aca1a59fda0f4. Latest consensus state: current_term: 1 leader_uuid: "6d1959434b934db4982aca1a59fda0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } } }
I20260808 04:53:02.815887 2362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:02.817796 2373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:02.818545 2363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d1959434b934db4982aca1a59fda0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } } }
I20260808 04:53:02.818740 2363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:02.819701 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:02.801559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.15.30.126:42569)
0808 04:53:02.802257 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:02.802283 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:02.802323 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:53:02.803269 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:02.803294 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:02.803707 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:02.803990 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:02.804938 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:02.806050 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:02.807477 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:02.808056 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:02.808176 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:02.811363 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:02.811368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:02.811401 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:02.811437 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:02.813713 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:02.814301 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:02.814307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:02.814313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:02.814449 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:02.814779 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:02.814782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:02.814784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:02.815299 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:02.815529 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:02.818219 (+ 2690us) server_negotiation.cc:299] Negotiation successful
0808 04:53:02.818514 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":168,"threads_started":1}
I20260808 04:53:02.821841 2373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:02.822620 2355 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:53:02.834071 2373 catalog_manager.cc:1347] Generated new cluster ID: b661a9d2d81142a9a27a3715f2f82490
I20260808 04:53:02.834152 2373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:02.851082 2373 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:53:02.852243 2373 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:02.858855 2373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Generated new TSK 0
I20260808 04:53:02.859515 2373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:53:03.715827 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42569
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:03.871017 2423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:03.871304 2423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:03.871388 2423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:03.875609 2423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:03.875692 2423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:03.875779 2423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:53:03.881417 2423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.30.84:56331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:03.882761 2423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:03.884086 2423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:03.897833 2431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:03.898200 2433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:03.898249 2423 server_base.cc:1034] running on GCE node
W20260808 04:53:03.898227 2435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:03.898978 2423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:03.899667 2423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:03.900861 2423 hybrid_clock.cc:648] HybridClock initialized: now 1786164783900830 us; error 47 us; skew 500 ppm
Aug 08 04:53:03 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164783, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:03.903976 2423 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:53:03.905038 2423 webserver.cc:460] Webserver started at http://127.15.30.65:38951/ using document root <none> and password file <none>
I20260808 04:53:03.905632 2423 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:03.905726 2423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:03.905947 2423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:53:03 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164783, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:03.922338 2423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "956c00fa18ab4d158f23289393ee2948"
format_stamp: "Formatted at 2026-08-08 04:53:03 on dist-test-slave-zpvx"
server_key: "34be547fc78d4d0269ac66cee03605dd"
server_key_iv: "8ee4a2dbb175bf124316deedffbddf7f"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:03.923166 2423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "956c00fa18ab4d158f23289393ee2948"
format_stamp: "Formatted at 2026-08-08 04:53:03 on dist-test-slave-zpvx"
server_key: "34be547fc78d4d0269ac66cee03605dd"
server_key_iv: "8ee4a2dbb175bf124316deedffbddf7f"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:03.928445 2423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:53:03.947751 2442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:03.949545 2423 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260808 04:53:03.949703 2423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "956c00fa18ab4d158f23289393ee2948"
format_stamp: "Formatted at 2026-08-08 04:53:03 on dist-test-slave-zpvx"
server_key: "34be547fc78d4d0269ac66cee03605dd"
server_key_iv: "8ee4a2dbb175bf124316deedffbddf7f"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:03.949849 2423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:03.966481 2423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:03.971000 2423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:03.971321 2423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:03.972232 2423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:03.973582 2423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:03.973707 2423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:03.973840 2423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:03.973922 2423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:03.992982 2423 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:42957
I20260808 04:53:03.994340 2423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:53:03.996551 2624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:42957 every 8 connection(s)
I20260808 04:53:03.998284 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 2423
I20260808 04:53:03.998420 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:53:03.998764 15481 external_mini_cluster.cc:1428] Setting key 1e947e55eda7672843864ce4ca1c2ff7
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164783, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:53:04.019517 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:03.997270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:40289 (local address 127.15.30.126:42569)
0808 04:53:03.997706 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:03.997711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:04.000327 (+ 2616us) server_negotiation.cc:407] Connection header received
0808 04:53:04.004456 (+ 4129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:04.004460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:04.004523 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:04.004615 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:04.006519 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.007405 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.009281 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.009669 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.009760 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:04.013642 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:04.013649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:04.013654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:04.013685 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:04.016152 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:04.017016 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:04.017022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:04.017025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:04.017149 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:04.017686 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:04.017690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:04.017692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:04.018006 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:04.018204 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:04.019181 (+ 977us) server_negotiation.cc:299] Negotiation successful
0808 04:53:04.019344 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
I20260808 04:53:04.021191 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:03.997768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42569 (local address 127.15.30.65:40289)
0808 04:53:03.999461 (+ 1693us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:03.999509 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:04.004048 (+ 4539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:04.004997 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:04.005007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:04.005505 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:04.006326 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:04.006350 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.007576 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.007579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:04.009126 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:04.009140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.009995 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.010000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:04.010238 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:04.011064 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:04.011097 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:04.013412 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:04.016321 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:04.016330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:04.016349 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:04.016824 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:04.017307 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:04.017312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:04.017316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:04.017532 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:04.018315 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:04.018331 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:04.018722 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 04:53:04.019793 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0808 04:53:04.020167 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"thread_start_us":198,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:53:04.022733 2625 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42569
I20260808 04:53:04.022989 2625 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:04.023597 2625 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:04.025458 2302 ts_manager.cc:194] Registered new tserver with Master: 956c00fa18ab4d158f23289393ee2948 (127.15.30.65:42957)
I20260808 04:53:04.027138 2302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:40289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:53:04.047540 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42569
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:53:04.204758 2639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:04.205049 2639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:04.205124 2639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:04.209972 2639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:04.210047 2639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:04.210130 2639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:53:04.216845 2639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.30.84:56331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:04.218225 2639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:04.219569 2639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:04.232897 2649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:04.232945 2655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:04.233410 2651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:04.234483 2639 server_base.cc:1034] running on GCE node
I20260808 04:53:04.234936 2639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:04.235687 2639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:04.238982 2639 hybrid_clock.cc:648] HybridClock initialized: now 1786164784238934 us; error 55 us; skew 500 ppm
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:04.242401 2639 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:53:04.243466 2639 webserver.cc:460] Webserver started at http://127.15.30.66:33731/ using document root <none> and password file <none>
I20260808 04:53:04.244127 2639 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:04.244237 2639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:04.244508 2639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:04.262807 2639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "651019bf731a4862ab67a0e9d992c45a"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "f9cb4dab6fb944ac4f9e8824ae13a888"
server_key_iv: "1c9ce41b7b6f661c2c0d56adf326beeb"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:04.263788 2639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "651019bf731a4862ab67a0e9d992c45a"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "f9cb4dab6fb944ac4f9e8824ae13a888"
server_key_iv: "1c9ce41b7b6f661c2c0d56adf326beeb"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:04.269987 2639 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 04:53:04.291455 2665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:04.293455 2639 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:53:04.293658 2639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "651019bf731a4862ab67a0e9d992c45a"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "f9cb4dab6fb944ac4f9e8824ae13a888"
server_key_iv: "1c9ce41b7b6f661c2c0d56adf326beeb"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:04.293848 2639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:04.324730 2639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:04.329021 2639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:04.329295 2639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:04.330252 2639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:04.331620 2639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:04.331707 2639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:04.331780 2639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:04.331815 2639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:04.354034 2639 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:34447
I20260808 04:53:04.354230 2833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:34447 every 8 connection(s)
I20260808 04:53:04.355548 2639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:53:04.364986 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 2639
I20260808 04:53:04.365165 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:53:04.365556 15481 external_mini_cluster.cc:1428] Setting key d3e1678145936e8665b4a20e843982a2
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:04.374953 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:04.358279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:50877 (local address 127.15.30.126:42569)
0808 04:53:04.358426 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:04.358432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:04.359849 (+ 1417us) server_negotiation.cc:407] Connection header received
0808 04:53:04.361187 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:04.361190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:04.361251 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:04.361376 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:04.363003 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.363936 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.365679 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.366064 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.366145 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:04.370433 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:04.370437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:04.370440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:04.370467 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:04.372325 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:04.373017 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:04.373021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:04.373022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:04.373128 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:04.373585 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:04.373589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:04.373591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:04.373820 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:04.373918 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:04.374647 (+ 729us) server_negotiation.cc:299] Negotiation successful
0808 04:53:04.374781 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:53:04.376288 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:04.358854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42569 (local address 127.15.30.66:50877)
0808 04:53:04.359675 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:04.359735 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:04.360913 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:04.361580 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:04.361589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:04.362061 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:04.362812 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:04.362833 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.364098 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.364102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:04.365531 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:04.365542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.366345 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.366349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:04.367243 (+ 894us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:04.368117 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:04.368137 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:04.370228 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:04.372469 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:04.372476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:04.372492 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:04.372877 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:04.373244 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:04.373247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:04.373249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:04.373447 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:04.373950 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:04.373957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:04.374336 (+ 379us) client_negotiation.cc:769] Sending connection context
0808 04:53:04.374556 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 04:53:04.374945 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":252,"threads_started":1}
I20260808 04:53:04.377993 2834 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42569
I20260808 04:53:04.378276 2834 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:04.378849 2834 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:04.380081 2302 ts_manager.cc:194] Registered new tserver with Master: 651019bf731a4862ab67a0e9d992c45a (127.15.30.66:34447)
I20260808 04:53:04.380952 2302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:50877
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:53:04.417397 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42569
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:04.596983 2842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:04.597442 2842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:04.597581 2842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:04.603678 2842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:04.603820 2842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:04.603951 2842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:53:04.611766 2842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.30.84:56331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:04.613385 2842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:04.614975 2842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:04.638171 2850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:04.638188 2848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:04.639628 2842 server_base.cc:1034] running on GCE node
W20260808 04:53:04.640566 2847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:04.641131 2842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:04.641925 2842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:04.649304 2842 hybrid_clock.cc:648] HybridClock initialized: now 1786164784649178 us; error 131 us; skew 500 ppm
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:04.654409 2842 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:53:04.655979 2842 webserver.cc:460] Webserver started at http://127.15.30.67:39441/ using document root <none> and password file <none>
I20260808 04:53:04.656903 2842 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:04.657009 2842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:04.657320 2842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:04.674183 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "56551bff28804ac2bf0727a0884252fd"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "6dd334d9718723a27e35e5b02391f4d2"
server_key_iv: "30b942ef78fc7772dba23be26ea6a75b"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:04.674988 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "56551bff28804ac2bf0727a0884252fd"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "6dd334d9718723a27e35e5b02391f4d2"
server_key_iv: "30b942ef78fc7772dba23be26ea6a75b"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:04.681022 2842 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 04:53:04.708905 2860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:04.710935 2842 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 04:53:04.711153 2842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "56551bff28804ac2bf0727a0884252fd"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "6dd334d9718723a27e35e5b02391f4d2"
server_key_iv: "30b942ef78fc7772dba23be26ea6a75b"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:04.711329 2842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:04.739944 2842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:04.744359 2842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:04.744808 2842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:04.745844 2842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:04.747181 2842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:04.747252 2842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:04.747325 2842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:04.747359 2842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:04.767956 2842 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:37301
I20260808 04:53:04.768949 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:37301 every 8 connection(s)
I20260808 04:53:04.770128 2842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:53:04.780098 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 2842
I20260808 04:53:04.780277 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:53:04.780704 15481 external_mini_cluster.cc:1428] Setting key 47f91ef35bad0988541fcf9a09bbdef8
Aug 08 04:53:04 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164784, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:04.785844 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:04.772293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:41621 (local address 127.15.30.126:42569)
0808 04:53:04.772476 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:04.772481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:04.773490 (+ 1009us) server_negotiation.cc:407] Connection header received
0808 04:53:04.774599 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:04.774603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:04.774662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:04.774742 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:04.776052 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.776998 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.778307 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.778536 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.778658 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:04.781810 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:04.781815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:04.781817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:04.781848 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:04.783579 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:04.784106 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:04.784110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:04.784112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:04.784200 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:04.784653 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:04.784657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:04.784658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:04.784857 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:04.784970 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:04.785545 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 04:53:04.785673 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:53:04.786806 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:04.772769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42569 (local address 127.15.30.67:41621)
0808 04:53:04.773332 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:04.773386 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:04.774368 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:04.774918 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:04.774925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:04.775303 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:04.775894 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:04.775910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.777179 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.777184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:04.778163 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:04.778174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:04.778704 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:04.778709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:04.778890 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:04.779588 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:04.779610 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:04.781616 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:04.783709 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:04.783714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:04.783727 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:04.783987 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:04.784303 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:04.784306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:04.784308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:04.784538 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:04.784970 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:04.784979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:04.785248 (+ 269us) client_negotiation.cc:769] Sending connection context
0808 04:53:04.785455 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:53:04.785806 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":174,"threads_started":1}
I20260808 04:53:04.787923 3037 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42569
I20260808 04:53:04.788164 3037 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:04.788753 3037 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:04.793208 2302 ts_manager.cc:194] Registered new tserver with Master: 56551bff28804ac2bf0727a0884252fd (127.15.30.67:37301)
I20260808 04:53:04.793941 2302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:41621
I20260808 04:53:04.795710 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:53:04.796551 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 2423
I20260808 04:53:04.833045 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 2639
I20260808 04:53:04.844792 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 2842
I20260808 04:53:04.858660 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 2253
I20260808 04:53:04.874876 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42569
--webserver_interface=127.15.30.126
--webserver_port=38097
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:42569
--ranger_config_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:04.987246 3043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:04.987495 3043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:04.987534 3043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:04.991505 3043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:04.991564 3043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:04.991583 3043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:04.991600 3043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:04.991614 3043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:04.996908 3043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:42569
--ranger_config_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=38097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:04.997915 3043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:04.998880 3043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:05.007913 3055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:05.008009 3043 server_base.cc:1034] running on GCE node
W20260808 04:53:05.007972 3052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:05.007913 3051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:05.008780 3043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:05.009910 3043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:05.011116 3043 hybrid_clock.cc:648] HybridClock initialized: now 1786164785011100 us; error 32 us; skew 500 ppm
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:05.013900 3043 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:05.014824 3043 webserver.cc:460] Webserver started at http://127.15.30.126:38097/ using document root <none> and password file <none>
I20260808 04:53:05.015385 3043 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:05.015486 3043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:05.066672 3043 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:53:05.069741 3069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.071455 3043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:53:05.071640 3043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "6d1959434b934db4982aca1a59fda0f4"
format_stamp: "Formatted at 2026-08-08 04:53:02 on dist-test-slave-zpvx"
server_key: "586caf0e518ba0680f94a719594c0fc5"
server_key_iv: "44267af0e0bf653c6387cc438a29f486"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:05.072453 3043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:05.119676 3043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:05.122394 3043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:05.122557 3043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:05.129953 3043 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:42569
I20260808 04:53:05.129966 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:42569 every 8 connection(s)
I20260808 04:53:05.131106 3043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:05.133351 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 3043
I20260808 04:53:05.133868 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:42957
--local_ip_for_outbound_sockets=127.15.30.65
--tserver_master_addrs=127.15.30.126:42569
--webserver_port=38951
--webserver_interface=127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:05.136322 3146 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:53:05.137974 3146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Bootstrap starting.
I20260808 04:53:05.155118 3146 log.cc:824] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:05.161991 3146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:53:05.162442 3146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Bootstrap complete.
I20260808 04:53:05.166689 3146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:05.167121 3146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1959434b934db4982aca1a59fda0f4, State: Initialized, Role: FOLLOWER
I20260808 04:53:05.167693 3146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:05.167852 3146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:05.167954 3146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:05.168150 3146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:53:05.170171 3146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:05.170516 3146 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1959434b934db4982aca1a59fda0f4; no voters:
I20260808 04:53:05.170855 3146 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:53:05.170941 3152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:53:05.171167 3152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [term 2 LEADER]: Becoming Leader. State: Replica: 6d1959434b934db4982aca1a59fda0f4, State: Running, Role: LEADER
I20260808 04:53:05.171458 3152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } }
I20260808 04:53:05.172125 3146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:05.172809 3153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1959434b934db4982aca1a59fda0f4. Latest consensus state: current_term: 2 leader_uuid: "6d1959434b934db4982aca1a59fda0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } } }
I20260808 04:53:05.172943 3153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:05.173571 3163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:05.173611 3154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6d1959434b934db4982aca1a59fda0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1959434b934db4982aca1a59fda0f4" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42569 } } }
I20260808 04:53:05.173738 3154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:05.176934 3163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:05.177485 3146 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:53:05.177650 3163 catalog_manager.cc:1259] Loaded cluster ID: b661a9d2d81142a9a27a3715f2f82490
I20260808 04:53:05.177755 3163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:05.180181 3163 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:05.183996 3163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Loaded TSK: 0
I20260808 04:53:05.185906 3163 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:53:05.281201 3150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:05.281522 3150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:05.281589 3150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:05.287995 3150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:05.288079 3150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:05.288168 3150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:53:05.295997 3150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:42957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=38951
--tserver_master_addrs=127.15.30.126:42569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:05.297304 3150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:05.298615 3150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:05.311410 3214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:05.311558 3150 server_base.cc:1034] running on GCE node
W20260808 04:53:05.311411 3209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:05.311973 3208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:05.312492 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:05.313284 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:05.314474 3150 hybrid_clock.cc:648] HybridClock initialized: now 1786164785314439 us; error 51 us; skew 500 ppm
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:05.318150 3150 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:53:05.319244 3150 webserver.cc:460] Webserver started at http://127.15.30.65:38951/ using document root <none> and password file <none>
I20260808 04:53:05.319957 3150 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:05.320047 3150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:05.346714 3150 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:53:05.350497 3224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.352109 3150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:53:05.352263 3150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "956c00fa18ab4d158f23289393ee2948"
format_stamp: "Formatted at 2026-08-08 04:53:03 on dist-test-slave-zpvx"
server_key: "34be547fc78d4d0269ac66cee03605dd"
server_key_iv: "8ee4a2dbb175bf124316deedffbddf7f"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:05.353050 3150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:05.374621 3150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:05.378400 3150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:05.378706 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:05.379506 3150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:05.380613 3150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:05.380681 3150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.380738 3150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:05.380769 3150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.392656 3150 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:42957
I20260808 04:53:05.392748 3416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:42957 every 8 connection(s)
I20260808 04:53:05.393699 3150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:53:05.397290 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 3150
I20260808 04:53:05.397825 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:34447
--local_ip_for_outbound_sockets=127.15.30.66
--tserver_master_addrs=127.15.30.126:42569
--webserver_port=33731
--webserver_interface=127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:05.419498 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:05.396148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42569 (local address 127.15.30.65:35809)
0808 04:53:05.396744 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:05.396796 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:05.398217 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:05.400309 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:05.400315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:05.400740 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:05.401411 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:05.401432 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.402947 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.402953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:05.404488 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:05.404504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.405946 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.405953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:05.406190 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:05.406913 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:05.406941 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:05.412710 (+ 5769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:05.415428 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:05.415434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:05.415449 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:05.415756 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:05.416222 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:05.416226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:05.416228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:05.416429 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:05.417454 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:05.417461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:05.417784 (+ 323us) client_negotiation.cc:769] Sending connection context
0808 04:53:05.417979 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 04:53:05.418418 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":167,"threads_started":1}
I20260808 04:53:05.421382 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:05.396028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35809 (local address 127.15.30.126:42569)
0808 04:53:05.398743 (+ 2715us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:05.398768 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:05.398815 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:53:05.399124 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:05.399141 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:05.399546 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:05.399925 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:05.401622 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.402800 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.404673 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.405487 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.405692 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:05.412959 (+ 7267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:05.412969 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:05.412987 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:05.413021 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:05.415282 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.415896 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.415902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.415908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.416101 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.416544 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.416548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.416550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.417244 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:05.417392 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:05.420099 (+ 2707us) server_negotiation.cc:299] Negotiation successful
0808 04:53:05.420447 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2523,"thread_start_us":127,"threads_started":1}
I20260808 04:53:05.424918 3417 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42569
I20260808 04:53:05.425168 3417 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.425724 3417 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
W20260808 04:53:05.427968 3417 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:42569 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:53:05.428084 3417 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.428247 3417 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:05.429168 3417 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.429335 3417 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
W20260808 04:53:05.430171 3417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:53:05.553812 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:05.554126 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:05.554193 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:05.558563 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:05.558632 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:05.558743 3421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:53:05.564273 3421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:34447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=33731
--tserver_master_addrs=127.15.30.126:42569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:05.565368 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:05.566474 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:05.577031 3429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:05.577061 3432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:05.577055 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:05.577260 3421 server_base.cc:1034] running on GCE node
I20260808 04:53:05.577728 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:05.578298 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:05.579455 3421 hybrid_clock.cc:648] HybridClock initialized: now 1786164785579419 us; error 57 us; skew 500 ppm
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:05.582301 3421 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:53:05.583192 3421 webserver.cc:460] Webserver started at http://127.15.30.66:33731/ using document root <none> and password file <none>
I20260808 04:53:05.583746 3421 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:05.583863 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:05.604100 3421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:53:05.613273 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.614926 3421 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.002s
I20260808 04:53:05.615072 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "651019bf731a4862ab67a0e9d992c45a"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "f9cb4dab6fb944ac4f9e8824ae13a888"
server_key_iv: "1c9ce41b7b6f661c2c0d56adf326beeb"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:05.615804 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:05.647162 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:05.651125 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:05.651333 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:05.652048 3421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:05.653120 3421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:05.653185 3421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.653244 3421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:05.653287 3421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.665518 3421 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:34447
I20260808 04:53:05.666679 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:53:05.667644 3624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:34447 every 8 connection(s)
I20260808 04:53:05.668972 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 3421
I20260808 04:53:05.669507 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.30.84:56331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:37301
--local_ip_for_outbound_sockets=127.15.30.67
--tserver_master_addrs=127.15.30.126:42569
--webserver_port=39441
--webserver_interface=127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:05.687997 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:05.668871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:32971 (local address 127.15.30.126:42569)
0808 04:53:05.669007 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:05.669012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:05.670209 (+ 1197us) server_negotiation.cc:407] Connection header received
0808 04:53:05.671180 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:05.671183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:05.671246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:05.671354 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:05.672515 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.673452 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.675623 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.676094 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.676203 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:05.679660 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:05.679666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:05.679672 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:05.679712 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:05.685211 (+ 5499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.685920 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.685924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.685926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.686020 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.686455 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.686458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.686460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.686693 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:05.686768 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:05.687671 (+ 903us) server_negotiation.cc:299] Negotiation successful
0808 04:53:05.687816 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:53:05.689733 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:05.669352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42569 (local address 127.15.30.66:32971)
0808 04:53:05.670046 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:05.670101 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:05.670962 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:05.671556 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:05.671563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:05.671840 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:05.672323 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:05.672338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.673950 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.673956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:05.675466 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:05.675482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.676462 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.676469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:05.676709 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:05.677438 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:05.677464 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:05.679438 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:05.685378 (+ 5940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:05.685388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:05.685407 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:05.685791 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:05.686146 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:05.686149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:05.686152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:05.686342 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:05.687053 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:05.687061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:05.687381 (+ 320us) client_negotiation.cc:769] Sending connection context
0808 04:53:05.688306 (+ 925us) client_negotiation.cc:240] Negotiation successful
0808 04:53:05.688682 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":143,"threads_started":1}
I20260808 04:53:05.691123 3625 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42569
I20260808 04:53:05.691361 3625 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.691901 3625 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
W20260808 04:53:05.693583 3625 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:42569 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:53:05.693703 3625 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.693894 3625 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:05.694782 3625 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.694934 3625 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
W20260808 04:53:05.695685 3625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:53:05.813735 3629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:05.814023 3629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:05.814100 3629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:05.818773 3629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:05.818856 3629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:05.818958 3629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:53:05.825068 3629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.30.84:56331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:37301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=39441
--tserver_master_addrs=127.15.30.126:42569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:05.826251 3629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:05.827699 3629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:05.840176 3639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:05.840308 3641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:05.840417 3638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:05.840994 3629 server_base.cc:1034] running on GCE node
I20260808 04:53:05.841419 3629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:05.842134 3629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:05.843326 3629 hybrid_clock.cc:648] HybridClock initialized: now 1786164785843289 us; error 47 us; skew 500 ppm
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:05.846415 3629 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:53:05.847487 3629 webserver.cc:460] Webserver started at http://127.15.30.67:39441/ using document root <none> and password file <none>
I20260808 04:53:05.848094 3629 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:05.848204 3629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:05.872990 3629 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:53:05.876536 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.878219 3629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:53:05.878399 3629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "56551bff28804ac2bf0727a0884252fd"
format_stamp: "Formatted at 2026-08-08 04:53:04 on dist-test-slave-zpvx"
server_key: "6dd334d9718723a27e35e5b02391f4d2"
server_key_iv: "30b942ef78fc7772dba23be26ea6a75b"
server_key_version: "kuduclusterkey@0"
I20260808 04:53:05.879014 3629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:05.893816 3629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:05.897501 3629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:05.897722 3629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:05.898492 3629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:05.899572 3629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:05.899642 3629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.899708 3629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:05.899740 3629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:05.918911 3629 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:37301
I20260808 04:53:05.918973 3833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:37301 every 8 connection(s)
I20260808 04:53:05.920670 3629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:53:05.930258 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 3629
Aug 08 04:53:05 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164785, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:05.939095 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:05.923123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:36531 (local address 127.15.30.126:42569)
0808 04:53:05.923313 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:05.923319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:05.924401 (+ 1082us) server_negotiation.cc:407] Connection header received
0808 04:53:05.925764 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:05.925768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:05.925830 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:05.925944 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:05.927411 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.928253 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.930012 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.930434 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.930591 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:05.934015 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:05.934036 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:05.934041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:05.934078 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:05.936456 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.937208 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.937212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.937213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.937318 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.937792 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.937797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.937799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.938030 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:05.938125 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:05.938802 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 04:53:05.938920 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:53:05.940018 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:05.923523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42569 (local address 127.15.30.67:36531)
0808 04:53:05.924216 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:05.924272 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:05.925484 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:05.926135 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:05.926142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:05.926445 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:05.927212 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:05.927235 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.928494 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.928496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:05.929795 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:05.929805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.930636 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.930641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:05.930906 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:05.931709 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:05.931731 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:05.933802 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:05.936595 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:05.936602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:05.936620 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:05.937026 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:05.937443 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:05.937446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:05.937448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:05.937654 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:05.938194 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:05.938200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:05.938542 (+ 342us) client_negotiation.cc:769] Sending connection context
0808 04:53:05.938741 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 04:53:05.939075 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":237,"threads_started":1}
I20260808 04:53:05.941355 3835 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42569
I20260808 04:53:05.941620 3835 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.942339 3835 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
W20260808 04:53:05.944103 3835 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:42569 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:53:05.944237 3835 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.944438 3835 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:05.945305 3835 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:05.945483 3835 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
W20260808 04:53:05.946419 3835 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:53:05.947839 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:05.931292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.15.30.126:42569)
0808 04:53:05.933283 (+ 1991us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:05.933287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:05.934347 (+ 1060us) server_negotiation.cc:407] Connection header received
0808 04:53:05.934410 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:05.934413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:05.934466 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:05.936478 (+ 2012us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:05.936596 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.937320 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.940483 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:05.940801 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:05.940913 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:05.943193 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:05.943201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:05.943224 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:05.943253 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:05.945901 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.946531 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.946535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.946537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.946626 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:05.946920 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:05.946924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:05.946926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:05.947097 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:05.947328 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:05.947528 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 04:53:05.947646 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1910,"thread_start_us":105,"threads_started":1}
I20260808 04:53:06.430425 3417 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:06.430752 3417 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:06.432067 3092 ts_manager.cc:194] Registered new tserver with Master: 956c00fa18ab4d158f23289393ee2948 (127.15.30.65:42957)
I20260808 04:53:06.433835 3092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:35809
Aug 08 04:53:06 dist-test-slave-zpvx krb5kdc[1605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786164786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:06.695921 3625 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:06.696362 3625 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:06.697364 3092 ts_manager.cc:194] Registered new tserver with Master: 651019bf731a4862ab67a0e9d992c45a (127.15.30.66:34447)
I20260808 04:53:06.698391 3092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:32971
Aug 08 04:53:06 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:06.946659 3835 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:06.946954 3835 heartbeater.cc:511] Master 127.15.30.126:42569 requested a full tablet report, sending...
I20260808 04:53:06.949020 3092 ts_manager.cc:194] Registered new tserver with Master: 56551bff28804ac2bf0727a0884252fd (127.15.30.67:37301)
I20260808 04:53:06.949769 3092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:36531
I20260808 04:53:06.962137 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:53:06.990839 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:06.968804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.15.30.126:42569)
0808 04:53:06.971119 (+ 2315us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:06.971124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:06.971140 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:53:06.971185 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:06.971188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:06.971244 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:06.971344 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:06.974083 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:06.976351 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:06.977525 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:06.977910 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:06.978004 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:06.986864 (+ 8860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:06.986868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:06.986872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:06.986905 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:06.988886 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:06.989415 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:06.989419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:06.989421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:06.989513 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:06.989852 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:06.989855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:06.989857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:06.990076 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:06.990216 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:06.990518 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 04:53:06.990651 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2235,"thread_start_us":189,"threads_started":1}
I20260808 04:53:06.995313 3092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:53:06.999727 3092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:53:07.019313 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.011447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37301 (local address 127.0.0.1:49400)
0808 04:53:07.012767 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:07.012812 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:07.013007 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:07.014193 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:07.014199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:07.014224 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:07.014570 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.014581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.016065 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.016068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.017492 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.017503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.018782 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.018785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.018901 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.018938 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:53:07.019040 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:53:07.019148 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":102,"threads_started":1}
I20260808 04:53:07.019887 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.011887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.15.30.67:37301)
0808 04:53:07.012215 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:07.012222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:07.012901 (+ 679us) server_negotiation.cc:407] Connection header received
0808 04:53:07.013183 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:07.013189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:07.013395 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:07.013509 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:07.014718 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.015920 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.017633 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.018463 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.018545 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.018683 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:07.019650 (+ 967us) server_negotiation.cc:299] Negotiation successful
0808 04:53:07.019751 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260808 04:53:07.022297 3884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.012799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:34447 (local address 127.0.0.1:43306)
0808 04:53:07.014709 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:07.014744 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:07.014880 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:07.015802 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:07.015805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:07.015823 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:07.015907 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.015913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.018356 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.018362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.019641 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.019670 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.021899 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.021903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.022007 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.022032 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:53:07.022101 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 04:53:07.022166 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1825,"thread_start_us":131,"threads_started":1}
I20260808 04:53:07.023191 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.013236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.15.30.66:34447)
0808 04:53:07.015172 (+ 1936us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:07.015179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:07.015200 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:53:07.015277 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:07.015282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:07.015483 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:07.015638 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:07.016068 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.018184 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.019892 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.020761 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.020879 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.022844 (+ 1965us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:07.022912 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 04:53:07.023000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1817,"thread_start_us":140,"threads_started":1}
I20260808 04:53:07.023411 3733 tablet_service.cc:1467] Processing CreateTablet for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b (DEFAULT_TABLE table=test-table [id=444eeee18f9c4819b7023d72438e4b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:07.024564 3733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:07.026232 3518 tablet_service.cc:1467] Processing CreateTablet for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b (DEFAULT_TABLE table=test-table [id=444eeee18f9c4819b7023d72438e4b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:07.027230 3518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:07.030160 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.013739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:42957 (local address 127.0.0.1:54422)
0808 04:53:07.020308 (+ 6569us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:07.020322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:07.020466 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:07.024912 (+ 4446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:07.024916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:07.024934 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:07.025014 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.025020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.027361 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.027364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.028491 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.028502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.029747 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.029751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.029858 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.029882 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:53:07.029935 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 04:53:07.029999 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6460,"thread_start_us":139,"threads_started":1}
I20260808 04:53:07.030844 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.013897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.15.30.65:42957)
0808 04:53:07.024441 (+ 10544us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:07.024448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:07.024475 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:53:07.024552 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:07.024557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:07.024751 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:07.025166 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:07.025374 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.027213 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.028666 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.029551 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.030369 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.030519 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:07.030569 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 04:53:07.030666 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":92,"threads_started":1}
I20260808 04:53:07.033543 3303 tablet_service.cc:1467] Processing CreateTablet for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b (DEFAULT_TABLE table=test-table [id=444eeee18f9c4819b7023d72438e4b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:07.034504 3303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:07.041368 3890 tablet_bootstrap.cc:492] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Bootstrap starting.
I20260808 04:53:07.043720 3893 tablet_bootstrap.cc:492] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948: Bootstrap starting.
I20260808 04:53:07.044029 3890 tablet_bootstrap.cc:654] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:07.044963 3890 log.cc:824] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:07.045703 3891 tablet_bootstrap.cc:492] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: Bootstrap starting.
I20260808 04:53:07.046089 3893 tablet_bootstrap.cc:654] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:07.046948 3893 log.cc:824] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:07.047755 3891 tablet_bootstrap.cc:654] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:07.048555 3891 log.cc:824] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:07.050761 3893 tablet_bootstrap.cc:492] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948: No bootstrap required, opened a new log
I20260808 04:53:07.050922 3890 tablet_bootstrap.cc:492] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: No bootstrap required, opened a new log
I20260808 04:53:07.050958 3893 ts_tablet_manager.cc:1397] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:53:07.051095 3890 ts_tablet_manager.cc:1397] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260808 04:53:07.054780 3893 raft_consensus.cc:348] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.055032 3893 raft_consensus.cc:374] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:07.054877 3890 raft_consensus.cc:348] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.055084 3893 raft_consensus.cc:729] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956c00fa18ab4d158f23289393ee2948, State: Initialized, Role: FOLLOWER
I20260808 04:53:07.055125 3890 raft_consensus.cc:374] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:07.055171 3890 raft_consensus.cc:729] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56551bff28804ac2bf0727a0884252fd, State: Initialized, Role: FOLLOWER
I20260808 04:53:07.055594 3890 consensus_queue.cc:260] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.055876 3893 consensus_queue.cc:260] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.056732 3890 ts_tablet_manager.cc:1428] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:53:07.057085 3835 heartbeater.cc:503] Master 127.15.30.126:42569 was elected leader, sending a full tablet report...
I20260808 04:53:07.057159 3893 ts_tablet_manager.cc:1428] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:53:07.057547 3417 heartbeater.cc:503] Master 127.15.30.126:42569 was elected leader, sending a full tablet report...
I20260808 04:53:07.059690 3891 tablet_bootstrap.cc:492] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: No bootstrap required, opened a new log
I20260808 04:53:07.059916 3891 ts_tablet_manager.cc:1397] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20260808 04:53:07.063786 3891 raft_consensus.cc:348] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.064021 3891 raft_consensus.cc:374] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:07.064085 3891 raft_consensus.cc:729] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651019bf731a4862ab67a0e9d992c45a, State: Initialized, Role: FOLLOWER
I20260808 04:53:07.064553 3891 consensus_queue.cc:260] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.065649 3891 ts_tablet_manager.cc:1428] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 04:53:07.066002 3625 heartbeater.cc:503] Master 127.15.30.126:42569 was elected leader, sending a full tablet report...
W20260808 04:53:07.147049 3418 tablet.cc:2369] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:53:07.169406 3626 tablet.cc:2369] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:53:07.172844 3839 tablet.cc:2369] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:53:07.180089 3903 raft_consensus.cc:482] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:07.180354 3903 raft_consensus.cc:504] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.181741 3903 leader_election.cc:288] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56551bff28804ac2bf0727a0884252fd (127.15.30.67:37301), 956c00fa18ab4d158f23289393ee2948 (127.15.30.65:42957)
I20260808 04:53:07.188625 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.182371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:41927 (local address 127.15.30.67:37301)
0808 04:53:07.182496 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:07.182502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:07.183179 (+ 677us) server_negotiation.cc:407] Connection header received
0808 04:53:07.183542 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:07.183546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:07.183642 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:07.183722 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:07.184513 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.185392 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.187349 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.187892 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.187981 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.188031 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:07.188423 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 04:53:07.188501 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:53:07.189273 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.182532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:47717 (local address 127.15.30.65:42957)
0808 04:53:07.182683 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:07.182689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:07.183404 (+ 715us) server_negotiation.cc:407] Connection header received
0808 04:53:07.183681 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:07.183684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:07.183732 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:07.183780 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:07.184584 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.185424 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.187555 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.188284 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.188452 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.188532 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:07.189044 (+ 512us) server_negotiation.cc:299] Negotiation successful
0808 04:53:07.189145 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:53:07.189324 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.181970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37301 (local address 127.15.30.66:41927)
0808 04:53:07.182920 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:07.183032 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:07.183317 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:07.183941 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:07.183943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:07.183962 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:07.184335 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.184345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.185543 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.185546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.187204 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.187216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.188122 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.188126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.188260 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.188283 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:53:07.188962 (+ 679us) client_negotiation.cc:240] Negotiation successful
0808 04:53:07.189050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":4,"thread_start_us":128,"threads_started":1}
I20260808 04:53:07.190021 3752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a5fb059aac4d5bac2ae3b9f9c28f3b" candidate_uuid: "651019bf731a4862ab67a0e9d992c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56551bff28804ac2bf0727a0884252fd" is_pre_election: true
I20260808 04:53:07.190210 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.182067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:42957 (local address 127.15.30.66:47717)
0808 04:53:07.183285 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:07.183301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:07.183539 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:07.183913 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:07.183916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:07.183932 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:07.184427 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.184439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.185584 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.185587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.187293 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:07.187306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.188658 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.188662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:07.188829 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.188863 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:53:07.189896 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0808 04:53:07.189986 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":45,"spinlock_wait_cycles":17664,"thread_start_us":156,"threads_started":1}
I20260808 04:53:07.190328 3752 raft_consensus.cc:2454] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651019bf731a4862ab67a0e9d992c45a in term 0.
I20260808 04:53:07.190850 3452 leader_election.cc:302] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56551bff28804ac2bf0727a0884252fd, 651019bf731a4862ab67a0e9d992c45a; no voters:
I20260808 04:53:07.190882 3337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a5fb059aac4d5bac2ae3b9f9c28f3b" candidate_uuid: "651019bf731a4862ab67a0e9d992c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956c00fa18ab4d158f23289393ee2948" is_pre_election: true
I20260808 04:53:07.191136 3903 raft_consensus.cc:2781] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:53:07.191215 3903 raft_consensus.cc:482] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:53:07.191152 3337 raft_consensus.cc:2454] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651019bf731a4862ab67a0e9d992c45a in term 0.
I20260808 04:53:07.191252 3903 raft_consensus.cc:3037] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:07.192790 3903 raft_consensus.cc:504] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.193121 3903 leader_election.cc:288] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [CANDIDATE]: Term 1 election: Requested vote from peers 56551bff28804ac2bf0727a0884252fd (127.15.30.67:37301), 956c00fa18ab4d158f23289393ee2948 (127.15.30.65:42957)
I20260808 04:53:07.193557 3337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a5fb059aac4d5bac2ae3b9f9c28f3b" candidate_uuid: "651019bf731a4862ab67a0e9d992c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956c00fa18ab4d158f23289393ee2948"
I20260808 04:53:07.193584 3752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17a5fb059aac4d5bac2ae3b9f9c28f3b" candidate_uuid: "651019bf731a4862ab67a0e9d992c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56551bff28804ac2bf0727a0884252fd"
I20260808 04:53:07.193684 3337 raft_consensus.cc:3037] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:07.193694 3752 raft_consensus.cc:3037] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:07.195035 3752 raft_consensus.cc:2454] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651019bf731a4862ab67a0e9d992c45a in term 1.
I20260808 04:53:07.195230 3337 raft_consensus.cc:2454] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651019bf731a4862ab67a0e9d992c45a in term 1.
I20260808 04:53:07.195335 3452 leader_election.cc:302] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56551bff28804ac2bf0727a0884252fd, 651019bf731a4862ab67a0e9d992c45a; no voters:
I20260808 04:53:07.195606 3903 raft_consensus.cc:2781] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:07.195827 3903 raft_consensus.cc:686] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 1 LEADER]: Becoming Leader. State: Replica: 651019bf731a4862ab67a0e9d992c45a, State: Running, Role: LEADER
I20260808 04:53:07.196137 3903 consensus_queue.cc:237] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } }
I20260808 04:53:07.199440 3092 catalog_manager.cc:5495] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a reported cstate change: term changed from 0 to 1, leader changed from <none> to 651019bf731a4862ab67a0e9d992c45a (127.15.30.66). New cstate: current_term: 1 leader_uuid: "651019bf731a4862ab67a0e9d992c45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "651019bf731a4862ab67a0e9d992c45a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 34447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 } health_report { overall_health: UNKNOWN } } }
I20260808 04:53:07.255512 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.250039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.15.30.66:34447)
0808 04:53:07.250181 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:07.250186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:07.250204 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:53:07.250294 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:07.250297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:07.250369 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:07.250502 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:53:07.250805 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.251618 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.253218 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:07.253529 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:07.253661 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:07.254035 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:53:07.254625 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:53:07.254754 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:07.255205 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 04:53:07.255289 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:53:07.262974 3337 raft_consensus.cc:1261] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 956c00fa18ab4d158f23289393ee2948 [term 1 FOLLOWER]: Refusing update from remote peer 651019bf731a4862ab67a0e9d992c45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:07.263321 3752 raft_consensus.cc:1261] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 1 FOLLOWER]: Refusing update from remote peer 651019bf731a4862ab67a0e9d992c45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:07.264504 3911 consensus_queue.cc:1035] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "956c00fa18ab4d158f23289393ee2948" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:07.264963 3911 consensus_queue.cc:1035] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56551bff28804ac2bf0727a0884252fd" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:07.273658 3915 mvcc.cc:205] Tried to move back new op lower bound from 7316130968621289472 to 7316130968358187008. Current Snapshot: MvccSnapshot[applied={T|T < 7316130968621289472}]
I20260808 04:53:07.277855 3920 mvcc.cc:205] Tried to move back new op lower bound from 7316130968621289472 to 7316130968358187008. Current Snapshot: MvccSnapshot[applied={T|T < 7316130968621289472}]
I20260808 04:53:07.286419 3092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:07.286582 3092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:07.289376 3916 mvcc.cc:205] Tried to move back new op lower bound from 7316130968621289472 to 7316130968358187008. Current Snapshot: MvccSnapshot[applied={T|T < 7316130968621289472}]
I20260808 04:53:07.289716 3092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d1959434b934db4982aca1a59fda0f4: Sending DeleteTablet for 3 replicas of tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b
I20260808 04:53:07.290845 3733 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:07 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:49400
I20260808 04:53:07.291432 3303 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:07 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:54422
I20260808 04:53:07.292629 3518 tablet_service.cc:1514] Processing DeleteTablet for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:07 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:43306
I20260808 04:53:07.294917 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 3150
I20260808 04:53:07.294992 3935 tablet_replica.cc:330] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: stopping tablet replica
I20260808 04:53:07.295297 3935 raft_consensus.cc:2229] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:07.295646 3935 raft_consensus.cc:2258] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:07.297283 3935 ts_tablet_manager.cc:1905] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 04:53:07 dist-test-slave-zpvx krb5kdc[1605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164786, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for HTTP/127.15.30.84@KRBTEST.COM
I20260808 04:53:07.314019 3935 ts_tablet_manager.cc:1918] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:07.314234 3935 log.cc:1196] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/17a5fb059aac4d5bac2ae3b9f9c28f3b
I20260808 04:53:07.314749 3935 ts_tablet_manager.cc:1939] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 56551bff28804ac2bf0727a0884252fd: Deleting consensus metadata
I20260808 04:53:07.316265 3073 catalog_manager.cc:4841] TS 56551bff28804ac2bf0727a0884252fd (127.15.30.67:37301): tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b (table test-table [id=444eeee18f9c4819b7023d72438e4b55]) successfully deleted
I20260808 04:53:07.328085 3938 tablet_replica.cc:330] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: stopping tablet replica
I20260808 04:53:07.329531 3938 raft_consensus.cc:2229] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:53:07.330107 3938 raft_consensus.cc:2258] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 04:53:07.335500 3074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:42957 (error 108)
I20260808 04:53:07.335544 3938 ts_tablet_manager.cc:1905] T 17a5fb059aac4d5bac2ae3b9f9c28f3b P 651019bf731a4862ab67a0e9d992c45a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:07.336469 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 3421
I20260808 04:53:07.348697 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.348205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:42957 (local address 127.0.0.1:54424)
0808 04:53:07.348473 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:07.348570 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260808 04:53:07.348867 3074 catalog_manager.cc:4584] TS 956c00fa18ab4d158f23289393ee2948 (127.15.30.65:42957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17a5fb059aac4d5bac2ae3b9f9c28f3b: Network error: Client connection negotiation failed: client connection to 127.15.30.65:42957: connect: Connection refused (error 111)
I20260808 04:53:07.355818 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 3629
I20260808 04:53:07.357863 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:07.356034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:34447 (local address 127.0.0.1:43316)
0808 04:53:07.357703 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:07.357764 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.66:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1554}
I20260808 04:53:07.372637 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 3043
I20260808 04:53:07.393421 15481 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:53:07.749379 15481 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:53:07.749508 15481 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:53:08.104018 15481 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:53:08.104 UTC [1651] LOG: received smart shutdown request
2026-08-08 04:53:08.108 UTC [1651] LOG: background worker "logical replication launcher" (PID 1666) exited with exit code 1
2026-08-08 04:53:08.109 UTC [1661] LOG: shutting down
2026-08-08 04:53:08.279 UTC [1651] LOG: database system is shut down
2026-08-08T04:53:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50173 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:53:08 dist-test-slave-zpvx krb5kdc[3959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:53:08 dist-test-slave-zpvx krb5kdc[3959](info): set up 2 sockets
Aug 08 04:53:08 dist-test-slave-zpvx krb5kdc[3959](info): commencing operation
krb5kdc: starting...
W20260808 04:53:09.662133 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:53:09 dist-test-slave-zpvx krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:53:09.786725 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43601
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:43601,127.15.30.125:43031,127.15.30.124:45423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:09.900261 3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:09.900520 3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:09.900560 3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:09.904348 3981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:09.904408 3981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:09.904428 3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:09.904444 3981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:09.904459 3981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:09.909715 3981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:43601,127.15.30.125:43031,127.15.30.124:45423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:09.910632 3981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:09.911556 3981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:09.919834 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:09.919831 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:09.919881 3987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:09.920234 3981 server_base.cc:1034] running on GCE node
I20260808 04:53:09.920876 3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:09.921908 3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:09.923138 3981 hybrid_clock.cc:648] HybridClock initialized: now 1786164789923118 us; error 34 us; skew 500 ppm
Aug 08 04:53:09 dist-test-slave-zpvx krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:09.926157 3981 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:09.926972 3981 webserver.cc:460] Webserver started at http://127.15.30.126:46407/ using document root <none> and password file <none>
I20260808 04:53:09.927486 3981 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:09.927546 3981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:09.927708 3981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:09.929720 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "beb72a64d61c4b30ba6441e0d8dd0f85"
format_stamp: "Formatted at 2026-08-08 04:53:09 on dist-test-slave-zpvx"
server_key: "30665dbb80d89d5411bae7228a4f9c90"
server_key_iv: "03a9a477cc89b9ee5797ad9a82a73280"
server_key_version: "encryptionkey@0"
I20260808 04:53:09.930248 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "beb72a64d61c4b30ba6441e0d8dd0f85"
format_stamp: "Formatted at 2026-08-08 04:53:09 on dist-test-slave-zpvx"
server_key: "30665dbb80d89d5411bae7228a4f9c90"
server_key_iv: "03a9a477cc89b9ee5797ad9a82a73280"
server_key_version: "encryptionkey@0"
I20260808 04:53:09.933856 3981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:53:09.936163 3997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:09.937434 3981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:53:09.937577 3981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "beb72a64d61c4b30ba6441e0d8dd0f85"
format_stamp: "Formatted at 2026-08-08 04:53:09 on dist-test-slave-zpvx"
server_key: "30665dbb80d89d5411bae7228a4f9c90"
server_key_iv: "03a9a477cc89b9ee5797ad9a82a73280"
server_key_version: "encryptionkey@0"
I20260808 04:53:09.937687 3981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:09.999007 3981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:10.002220 3981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:10.002447 3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:10.009757 3981 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:43601
I20260808 04:53:10.009788 4078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:43601 every 8 connection(s)
I20260808 04:53:10.010718 3981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:10.013486 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 3981
I20260808 04:53:10.013530 4079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:10.013615 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:53:10.013931 15481 external_mini_cluster.cc:1428] Setting key 1a4c7791aaf2b77e3b90cd08a065b6ba
I20260808 04:53:10.018255 4079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.30.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.125@KRBTEST.COM" created.
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:10.035076 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.021156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:43601 (local address 127.0.0.1:54464)
0808 04:53:10.021714 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.021749 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.022551 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.023229 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.023234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.023675 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.023960 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.023975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.025755 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.025761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.026600 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.026615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.027325 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.027329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.027446 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.028045 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.028062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.029744 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.032139 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.032153 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.032161 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.032456 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.032857 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.032859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.032868 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.032982 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.033763 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.033768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.033889 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.034032 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.034303 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":155,"threads_started":1}
I20260808 04:53:10.035094 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.021127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.15.30.126:43601)
0808 04:53:10.021565 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.021583 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.021856 (+ 273us) server_negotiation.cc:407] Connection header received
0808 04:53:10.022859 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.022863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.023036 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.023225 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.024107 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.025614 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.026735 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.027168 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.027338 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.029904 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.029914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.029930 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.029964 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.032005 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.032575 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.032584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.032587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.032743 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.033089 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.033095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.033098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.033629 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.033783 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.034161 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.034414 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":295,"thread_start_us":147,"threads_started":1}
I20260808 04:53:10.037369 4079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:53:10.038086 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.037821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57884)
0808 04:53:10.037946 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.038017 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:53:10.038327 3999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
I20260808 04:53:10.039287 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.038993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57892)
0808 04:53:10.039134 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.039203 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 04:53:10.039471 4079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:43031: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab.
Entry for principal kudu/127.15.30.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab.
I20260808 04:53:10.043502 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:43031
--webserver_interface=127.15.30.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab
--principal=kudu/127.15.30.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:43601,127.15.30.125:43031,127.15.30.124:45423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:10.088763 4079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } attempt: 1
I20260808 04:53:10.089826 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.089493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57906)
0808 04:53:10.089670 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.089737 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 04:53:10.090618 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.090373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57908)
0808 04:53:10.090511 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.090558 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:53:10.090756 4079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:43031: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
W20260808 04:53:10.155284 4088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:10.155568 4088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:10.155646 4088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 04:53:10.158900 4079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } attempt: 2
I20260808 04:53:10.159803 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.159447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57920)
0808 04:53:10.159633 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.159710 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 04:53:10.159956 4088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:10.160029 4088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:10.160050 4088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:10.160068 4088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:10.160084 4088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:10.160668 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.160425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57934)
0808 04:53:10.160540 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.160594 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:53:10.160856 4079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:43031: Network error: Client connection negotiation failed: client connection to 127.15.30.125:43031: connect: Connection refused (error 111)
I20260808 04:53:10.165486 4088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:43601,127.15.30.125:43031,127.15.30.124:45423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab
--principal=kudu/127.15.30.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:43031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.30.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:10.166473 4088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:10.167480 4088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:10.175999 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:10.176043 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:10.176010 4088 server_base.cc:1034] running on GCE node
W20260808 04:53:10.176211 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:10.176864 4088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:10.177969 4088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:10.179164 4088 hybrid_clock.cc:648] HybridClock initialized: now 1786164790179145 us; error 31 us; skew 500 ppm
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:10.181926 4088 init.cc:377] Logged in from keytab as kudu/127.15.30.125@KRBTEST.COM (short username kudu)
I20260808 04:53:10.182766 4088 webserver.cc:460] Webserver started at http://127.15.30.125:41639/ using document root <none> and password file <none>
I20260808 04:53:10.183331 4088 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:10.183424 4088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:10.183660 4088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:10.185710 4088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data/instance:
uuid: "00beb1e70f4a47679f825bd15c2e17df"
format_stamp: "Formatted at 2026-08-08 04:53:10 on dist-test-slave-zpvx"
server_key: "b2e75e2c98c0101ff1250842bfd49116"
server_key_iv: "c11b83a3ea928d22ac3e55fb0133f11b"
server_key_version: "encryptionkey@0"
I20260808 04:53:10.186295 4088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/wal/instance:
uuid: "00beb1e70f4a47679f825bd15c2e17df"
format_stamp: "Formatted at 2026-08-08 04:53:10 on dist-test-slave-zpvx"
server_key: "b2e75e2c98c0101ff1250842bfd49116"
server_key_iv: "c11b83a3ea928d22ac3e55fb0133f11b"
server_key_version: "encryptionkey@0"
I20260808 04:53:10.190127 4088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260808 04:53:10.192624 4109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:10.193830 4088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:53:10.193974 4088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/wal
uuid: "00beb1e70f4a47679f825bd15c2e17df"
format_stamp: "Formatted at 2026-08-08 04:53:10 on dist-test-slave-zpvx"
server_key: "b2e75e2c98c0101ff1250842bfd49116"
server_key_iv: "c11b83a3ea928d22ac3e55fb0133f11b"
server_key_version: "encryptionkey@0"
I20260808 04:53:10.194089 4088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:10.210848 4088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:10.213688 4088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:10.213891 4088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:10.221597 4088 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.125:43031
I20260808 04:53:10.221628 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.125:43031 every 8 connection(s)
I20260808 04:53:10.222496 4088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
I20260808 04:53:10.225147 4184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:10.228770 4184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:53:10.230198 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 4088
I20260808 04:53:10.230345 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-1/wal/instance
I20260808 04:53:10.230698 15481 external_mini_cluster.cc:1428] Setting key 98cd7406b2ea3a35db0f226895febb3c
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.124@KRBTEST.COM" created.
I20260808 04:53:10.244323 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.230968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.15.30.126:43601)
0808 04:53:10.231201 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.231208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.231978 (+ 770us) server_negotiation.cc:407] Connection header received
0808 04:53:10.233309 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.233314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.233380 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.233458 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.234808 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.235688 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.236981 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.237463 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.237608 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.240132 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.240138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.240142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.240173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.241923 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.242550 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.242553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.242554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.242635 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.243027 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.243030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.243031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.243338 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.243454 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.243997 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.244143 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260808 04:53:10.245229 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.231191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:43601 (local address 127.0.0.1:54466)
0808 04:53:10.231776 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.231838 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.233093 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.233753 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.233762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.234185 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.234641 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.234654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.235868 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.235871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.236835 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.236846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.237641 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.237645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.237789 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.238448 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.238478 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.239946 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.242042 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.242047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.242059 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.242403 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.242752 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.242755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.242757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.242904 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.243464 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.243470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.243721 (+ 251us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.243928 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.244252 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":146,"threads_started":1}
I20260808 04:53:10.246899 4184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:53:10.252063 4079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } attempt: 3
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
I20260808 04:53:10.260131 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.247467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57948)
0808 04:53:10.247605 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.247617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.247682 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.248452 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.248456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.248651 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.248788 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.248799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.249796 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.249800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.250724 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.250738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.251376 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.251381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.251507 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.251967 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.251987 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.255257 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.257478 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.257484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.257487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.257812 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.258834 (+ 1022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.258838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.258840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.258930 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.259586 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.259588 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.259668 (+ 80us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.259785 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.259926 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:53:10.260334 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.247663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.15.30.125:43031)
0808 04:53:10.248079 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.248087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.248108 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:53:10.248143 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.248147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.248288 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.248456 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.248922 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.249666 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.250867 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.251241 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.251329 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.255426 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.255435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.255446 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.255466 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.257349 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.257898 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.257904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.257906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.258726 (+ 820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.259056 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.259062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.259065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.259416 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.259635 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.259878 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.260100 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab.
Entry for principal kudu/127.15.30.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab.
I20260808 04:53:10.261794 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:45423
--webserver_interface=127.15.30.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab
--principal=kudu/127.15.30.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:43601,127.15.30.125:43031,127.15.30.124:45423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:10.261834 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.252586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57952)
0808 04:53:10.252739 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.252751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.252821 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.253117 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.253119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.253298 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.253398 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.253403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.254298 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.254305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.255084 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.255094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.255585 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.255588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.255711 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.256209 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.256222 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.257718 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.259825 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.259828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.259830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.260099 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.260696 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.260699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.260701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.260798 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.261297 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.261301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.261408 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.261500 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.261655 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:53:10.262208 4184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:53:10.262887 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.262626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:49940)
0808 04:53:10.262736 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.262815 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:53:10.263093 4110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
I20260808 04:53:10.263170 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.252630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.15.30.125:43031)
0808 04:53:10.252930 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.252936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.252948 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:53:10.252976 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.252978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.253009 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.253080 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.253504 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.254181 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.255226 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.255463 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.255570 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.257853 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.257859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.257862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.257889 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.259698 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.260323 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.260329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.260332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.260560 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.260912 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.260918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.260923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.261160 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.262566 (+ 1406us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.262654 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.262802 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":125,"threads_started":1}
I20260808 04:53:10.264313 4079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:53:10.264424 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.264168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:49952)
0808 04:53:10.264271 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.264321 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:53:10.264746 4184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:45423: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
I20260808 04:53:10.265097 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.264798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:49956)
0808 04:53:10.264940 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.265008 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 04:53:10.265866 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.265533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:49968)
0808 04:53:10.265718 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.265781 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260808 04:53:10.266028 4079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:45423: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
I20260808 04:53:10.297188 4079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } attempt: 1
I20260808 04:53:10.298063 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.297711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:49972)
0808 04:53:10.297873 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.297970 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:53:10.299291 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.299039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:49986)
0808 04:53:10.299162 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.299222 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 04:53:10.299451 4079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:45423: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
I20260808 04:53:10.314006 4184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } attempt: 1
I20260808 04:53:10.314764 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.314470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:49992)
0808 04:53:10.314622 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.314684 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:53:10.315583 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.315311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50000)
0808 04:53:10.315457 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.315517 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 04:53:10.315754 4184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:45423: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
I20260808 04:53:10.374625 4079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } attempt: 2
I20260808 04:53:10.375499 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.375161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50016)
0808 04:53:10.375335 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.375407 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 04:53:10.376250 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.376023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50018)
0808 04:53:10.376130 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.376180 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:53:10.376405 4079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:45423: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
W20260808 04:53:10.377066 4197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:10.377287 4197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:10.377379 4197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:10.381408 4197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:10.381457 4197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:10.381476 4197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:10.381523 4197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:10.381552 4197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:10.383925 4184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } attempt: 2
I20260808 04:53:10.384660 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.384339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50034)
0808 04:53:10.384522 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.384580 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 04:53:10.385425 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.385200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50036)
0808 04:53:10.385310 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.385362 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:53:10.385572 4184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:45423: Network error: Client connection negotiation failed: client connection to 127.15.30.124:45423: connect: Connection refused (error 111)
I20260808 04:53:10.387279 4197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:43601,127.15.30.125:43031,127.15.30.124:45423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab
--principal=kudu/127.15.30.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:45423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.30.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:10.388360 4197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:10.389474 4197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:10.397583 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:10.397639 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:10.397635 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:10.398358 4197 server_base.cc:1034] running on GCE node
I20260808 04:53:10.398921 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:10.399897 4197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:10.401087 4197 hybrid_clock.cc:648] HybridClock initialized: now 1786164790401072 us; error 25 us; skew 500 ppm
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:10.403843 4197 init.cc:377] Logged in from keytab as kudu/127.15.30.124@KRBTEST.COM (short username kudu)
I20260808 04:53:10.404793 4197 webserver.cc:460] Webserver started at http://127.15.30.124:40959/ using document root <none> and password file <none>
I20260808 04:53:10.405344 4197 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:10.405437 4197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:10.405671 4197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:10.407647 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data/instance:
uuid: "bbf22b09a8104149a8c6a493a48bb631"
format_stamp: "Formatted at 2026-08-08 04:53:10 on dist-test-slave-zpvx"
server_key: "b919dd498b466dbde6e4e552d0b4588f"
server_key_iv: "b5c086b2dd831e95542a222775847874"
server_key_version: "encryptionkey@0"
I20260808 04:53:10.408494 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/wal/instance:
uuid: "bbf22b09a8104149a8c6a493a48bb631"
format_stamp: "Formatted at 2026-08-08 04:53:10 on dist-test-slave-zpvx"
server_key: "b919dd498b466dbde6e4e552d0b4588f"
server_key_iv: "b5c086b2dd831e95542a222775847874"
server_key_version: "encryptionkey@0"
I20260808 04:53:10.412324 4197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:53:10.414672 4225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:10.415899 4197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:10.415998 4197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/wal
uuid: "bbf22b09a8104149a8c6a493a48bb631"
format_stamp: "Formatted at 2026-08-08 04:53:10 on dist-test-slave-zpvx"
server_key: "b919dd498b466dbde6e4e552d0b4588f"
server_key_iv: "b5c086b2dd831e95542a222775847874"
server_key_version: "encryptionkey@0"
I20260808 04:53:10.416076 4197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:10.425397 4197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:10.428545 4197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:10.428691 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:10.435840 4197 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.124:45423
I20260808 04:53:10.435873 4304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.124:45423 every 8 connection(s)
I20260808 04:53:10.436820 4197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
I20260808 04:53:10.439302 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 4197
I20260808 04:53:10.439428 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786164620544947-15481-0/minicluster-data/master-2/wal/instance
I20260808 04:53:10.439610 4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:10.439757 15481 external_mini_cluster.cc:1428] Setting key 9333f763a16c4797cccecf78fa9e72a5
I20260808 04:53:10.444813 4305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:10.455272 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.441175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.15.30.124:45423)
0808 04:53:10.441800 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.441819 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.441873 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:53:10.442554 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.442576 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.443012 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.443374 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.444050 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.445539 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.446388 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.447044 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.447692 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.450322 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.450327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.450335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.450362 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.452207 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.452767 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.452773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.452785 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.452896 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.453235 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.453241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.453246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.453764 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.453977 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.454281 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.454570 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":118,"threads_started":1}
I20260808 04:53:10.456535 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.446680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.15.30.126:43601)
0808 04:53:10.446833 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.446838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.447168 (+ 330us) server_negotiation.cc:407] Connection header received
0808 04:53:10.447317 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.447319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.447364 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.447433 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.448238 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.449093 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.450235 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.450460 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.450539 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.453012 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.453016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.453018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.453052 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.454677 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.455163 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.455167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.455169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.455254 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.455566 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.455570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.455571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.455762 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.455874 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.456240 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.456352 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:53:10.456548 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.446614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:43601 (local address 127.0.0.1:54480)
0808 04:53:10.447034 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.447084 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.447211 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.447527 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.447535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.447958 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.448098 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.448104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.449233 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.449237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.450098 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.450109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.450588 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.450594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.450703 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.451135 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.451153 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.452868 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.454806 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.454818 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.454828 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.455047 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.455350 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.455354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.455356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.455472 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.455878 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.455887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.455998 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.456143 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.456352 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":160,"threads_started":1}
I20260808 04:53:10.457901 4305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
I20260808 04:53:10.466295 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.457562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.15.30.126:43601)
0808 04:53:10.457665 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.457669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.457711 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:53:10.457862 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.457864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.457901 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.457964 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.458692 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.459134 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.460082 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.460308 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.460449 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.462878 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.462881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.462884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.462908 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.464606 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.465040 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.465042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.465043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.465146 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.465456 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.465463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.465464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.465681 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.465907 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.465992 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.466115 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:53:10.466647 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.458422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:43031 (local address 127.0.0.1:57964)
0808 04:53:10.458560 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.458589 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.458664 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.458913 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.458916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.459081 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.459169 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.459196 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.460080 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.460085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.460810 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.460820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.461381 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.461387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.461520 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.461913 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.461933 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.463114 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.464967 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.464973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.464975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.465228 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.465506 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.465509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.465510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.465586 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.466024 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.466027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.466130 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.466313 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.466478 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 04:53:10.466674 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.458541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.15.30.125:43031)
0808 04:53:10.458684 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.458690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.458707 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:53:10.458769 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.458771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.458811 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.458898 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.459327 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.459963 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.461002 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.461252 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.461337 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.463292 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.463296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.463299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.463330 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.464861 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.465316 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.465319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.465320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.465389 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.465681 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.465685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.465687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.465910 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.466012 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.466389 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.466520 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:53:10.467429 4305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
I20260808 04:53:10.475600 4079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } attempt: 3
I20260808 04:53:10.475808 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.468019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.15.30.124:45423)
0808 04:53:10.468108 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.468114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.468161 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:53:10.468266 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.468269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.468308 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.468418 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.468751 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.469601 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.470316 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.470543 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.470647 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.472496 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.472499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.472501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.472522 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.474121 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.474533 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.474538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.474540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.474615 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.474927 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.474933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.474938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.475126 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.475254 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.475534 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.475665 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:53:10.475844 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.467967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50066)
0808 04:53:10.468107 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.468121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.468197 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.468424 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.468426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.468582 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.468637 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.468641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.469716 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.469722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.470202 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.470208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.470658 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.470662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.470751 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.471163 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.471174 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.472298 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.474252 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.474255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.474257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.474436 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.474749 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.474751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.474752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.474831 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.475262 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.475264 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.475361 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.475474 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.475596 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:53:10.476740 4184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } attempt: 3
I20260808 04:53:10.477010 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.467501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.15.30.125:43031)
0808 04:53:10.467597 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.467601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.467620 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:53:10.467720 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.467722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.467753 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.467853 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.468500 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.468981 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.469745 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.469971 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.470080 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.473022 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.473026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.473029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.473054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.474578 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.475009 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.475012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.475013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.475093 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.475389 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.475393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.475395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.475615 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.475806 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.476487 (+ 681us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.476625 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 04:53:10.480137 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:53:10.480784 4305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: Bootstrap starting.
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164789, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
Aug 08 04:53:10 dist-test-slave-zpvx krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
I20260808 04:53:10.483992 4305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:10.484985 4305 log.cc:824] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:10.486030 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.475988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50078)
0808 04:53:10.476110 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.476126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.476212 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.476759 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.476763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.476938 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.477036 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.477046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.478071 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.478075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.478796 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.478807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.479698 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.479702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.479779 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.480247 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.480261 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.481702 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.484206 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.484212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.484216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.484504 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.484841 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.484845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.484849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.484947 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.485471 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.485474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.485572 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.485713 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.485851 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:53:10.486052 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.476237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.15.30.124:45423)
0808 04:53:10.476332 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.476337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.476353 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:53:10.476426 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.476428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.476463 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.476553 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.477284 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.477946 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.479255 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.479570 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.479986 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.481855 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.481858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.481861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.481886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.484094 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.484628 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.484634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.484640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.484725 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.485065 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.485069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.485070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.485276 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.485385 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.485771 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.485890 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:53:10.487514 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.477130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:45423 (local address 127.0.0.1:50084)
0808 04:53:10.477228 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:10.477241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:10.477310 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:10.478126 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:10.478129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:10.478317 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:10.478417 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.478427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.479499 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.479503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.480277 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:10.480289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.481013 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.481016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:10.481097 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.481499 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:10.481513 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:10.482866 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:10.485895 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.485898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.485900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.486133 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.486435 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:10.486440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:10.486443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:10.486538 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:10.486975 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:10.486980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:10.487066 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 04:53:10.487159 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:53:10.487313 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 04:53:10.488049 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.477569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.15.30.124:45423)
0808 04:53:10.477889 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.477898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.477910 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:53:10.477941 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.477943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.477979 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.478536 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.478640 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.479367 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.480498 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.480826 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.481304 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.483021 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.483026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.483029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.483056 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.485768 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.486263 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.486266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.486270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.486353 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.486657 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.486660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.486662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.486867 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.487679 (+ 812us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.487763 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.487874 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260808 04:53:10.489279 4079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: Bootstrap starting.
I20260808 04:53:10.490602 4305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: No bootstrap required, opened a new log
I20260808 04:53:10.491885 4079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:10.494810 4305 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.495092 4305 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:10.495159 4305 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf22b09a8104149a8c6a493a48bb631, State: Initialized, Role: FOLLOWER
I20260808 04:53:10.495668 4305 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.495957 4184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df: Bootstrap starting.
I20260808 04:53:10.496754 4079 log.cc:824] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:10.497012 4305 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:10.498855 4184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:10.502028 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.484063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.15.30.126:43601)
0808 04:53:10.484193 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.484199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.492465 (+ 8266us) server_negotiation.cc:407] Connection header received
0808 04:53:10.492509 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.492512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.492558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.492632 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.493716 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.494402 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.495738 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.496070 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.496152 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.498033 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.498036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.498039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.498065 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.500076 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.500621 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.500627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.500631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.500720 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.501033 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.501038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.501043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.501272 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.501381 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.501739 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.501857 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 04:53:10.502450 4346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:53:10.502537 4346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:53:10.500686 4197 master_runner.cc:186] Error getting master registration for 127.15.30.126:43601: OK, instance_id { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" instance_seqno: 1786164790007590 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:53:10.503852 4197 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:53:10.504127 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.483046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.15.30.124:45423)
0808 04:53:10.483702 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.483710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.485938 (+ 2228us) server_negotiation.cc:407] Connection header received
0808 04:53:10.485976 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.485979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.486018 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.486111 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.487390 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.488178 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.492660 (+ 4482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.493067 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.493178 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.495857 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.495861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.495863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.495887 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.499790 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.500806 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.500809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.500812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.500899 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.501204 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.501207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.501209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.501414 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.501524 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.503060 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.503941 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":106,"threads_started":1}
I20260808 04:53:10.504537 4079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: No bootstrap required, opened a new log
I20260808 04:53:10.505524 4184 log.cc:824] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:10.506597 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:10.483663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.15.30.125:43031)
0808 04:53:10.483764 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:10.483768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:10.486758 (+ 2990us) server_negotiation.cc:407] Connection header received
0808 04:53:10.486793 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:10.486796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:10.486835 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:10.486903 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:10.487809 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.488472 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.493717 (+ 5245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:10.494033 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:10.494124 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:10.502024 (+ 7900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:10.502028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:10.502030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:10.502060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:10.504132 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.504938 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.504944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.504945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.505032 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:10.505337 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:10.505341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:10.505342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:10.505545 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:10.505726 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:10.506344 (+ 618us) server_negotiation.cc:299] Negotiation successful
0808 04:53:10.506454 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:53:10.507483 4184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df: No bootstrap required, opened a new log
I20260808 04:53:10.508046 4325 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.508330 4325 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:53:10.508693 4079 raft_consensus.cc:348] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.508905 4079 raft_consensus.cc:374] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:10.508960 4079 raft_consensus.cc:729] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beb72a64d61c4b30ba6441e0d8dd0f85, State: Initialized, Role: FOLLOWER
I20260808 04:53:10.509438 4079 consensus_queue.cc:260] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.509953 4351 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.510141 4351 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:53:10.510747 4079 sys_catalog.cc:564] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:10.511385 4184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.511634 4184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:10.511705 4184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00beb1e70f4a47679f825bd15c2e17df, State: Initialized, Role: FOLLOWER
I20260808 04:53:10.512200 4184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.512730 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.514532 4184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:10.514645 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:53:10.515390 3981 master_runner.cc:186] Error getting master registration for 127.15.30.125:43031: OK, instance_id { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" instance_seqno: 1786164790219372 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:53:10.519726 3981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:53:10.520035 4389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:53:10.520143 4389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 04:53:10.524137 4370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:53:10.524255 4370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:53:10.524706 4088 master_runner.cc:263] No leader master found from master 00beb1e70f4a47679f825bd15c2e17df
I20260808 04:53:10.524778 4088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:53:10.558511 4365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:10.558727 4365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.559783 4365 leader_election.cc:288] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beb72a64d61c4b30ba6441e0d8dd0f85 (127.15.30.126:43601), bbf22b09a8104149a8c6a493a48bb631 (127.15.30.124:45423)
I20260808 04:53:10.560510 4272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00beb1e70f4a47679f825bd15c2e17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf22b09a8104149a8c6a493a48bb631" is_pre_election: true
I20260808 04:53:10.560839 4272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00beb1e70f4a47679f825bd15c2e17df in term 0.
I20260808 04:53:10.561232 4110 leader_election.cc:302] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00beb1e70f4a47679f825bd15c2e17df, bbf22b09a8104149a8c6a493a48bb631; no voters:
I20260808 04:53:10.562085 4039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00beb1e70f4a47679f825bd15c2e17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" is_pre_election: true
I20260808 04:53:10.562355 4039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00beb1e70f4a47679f825bd15c2e17df in term 0.
I20260808 04:53:10.562932 4365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:53:10.563024 4365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:53:10.563076 4365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:10.564460 4365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.564806 4365 leader_election.cc:288] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [CANDIDATE]: Term 1 election: Requested vote from peers beb72a64d61c4b30ba6441e0d8dd0f85 (127.15.30.126:43601), bbf22b09a8104149a8c6a493a48bb631 (127.15.30.124:45423)
I20260808 04:53:10.565392 4039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00beb1e70f4a47679f825bd15c2e17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85"
I20260808 04:53:10.565529 4039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:10.566811 4039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00beb1e70f4a47679f825bd15c2e17df in term 1.
I20260808 04:53:10.567126 4272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00beb1e70f4a47679f825bd15c2e17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf22b09a8104149a8c6a493a48bb631"
I20260808 04:53:10.567256 4272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:10.568524 4272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00beb1e70f4a47679f825bd15c2e17df in term 1.
I20260808 04:53:10.568840 4110 leader_election.cc:302] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00beb1e70f4a47679f825bd15c2e17df, bbf22b09a8104149a8c6a493a48bb631; no voters:
I20260808 04:53:10.571363 4365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:10.571465 4365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [term 1 LEADER]: Becoming Leader. State: Replica: 00beb1e70f4a47679f825bd15c2e17df, State: Running, Role: LEADER
I20260808 04:53:10.571803 4365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } }
I20260808 04:53:10.573623 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00beb1e70f4a47679f825bd15c2e17df. Latest consensus state: current_term: 1 leader_uuid: "00beb1e70f4a47679f825bd15c2e17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.573740 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:10.574149 4396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:10.575121 4396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:10.584949 4272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [term 1 FOLLOWER]: Refusing update from remote peer 00beb1e70f4a47679f825bd15c2e17df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:10.585605 4393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:10.590423 4325 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00beb1e70f4a47679f825bd15c2e17df. Latest consensus state: current_term: 1 leader_uuid: "00beb1e70f4a47679f825bd15c2e17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.590553 4325 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:53:10.591481 4039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [term 1 FOLLOWER]: Refusing update from remote peer 00beb1e70f4a47679f825bd15c2e17df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:10.592090 4393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [LEADER]: Connected to new peer: Peer: permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:10.596149 4351 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00beb1e70f4a47679f825bd15c2e17df. Latest consensus state: current_term: 1 leader_uuid: "00beb1e70f4a47679f825bd15c2e17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.596284 4351 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:53:10.603271 4393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00beb1e70f4a47679f825bd15c2e17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.603403 4393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:10.604610 4351 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00beb1e70f4a47679f825bd15c2e17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.604722 4351 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:53:10.605074 4398 mvcc.cc:205] Tried to move back new op lower bound from 7316130982215720960 to 7316130982185263104. Current Snapshot: MvccSnapshot[applied={T|T < 7316130982215720960}]
I20260808 04:53:10.605453 4405 mvcc.cc:205] Tried to move back new op lower bound from 7316130982215720960 to 7316130982185263104. Current Snapshot: MvccSnapshot[applied={T|T < 7316130982215720960}]
I20260808 04:53:10.608703 4396 catalog_manager.cc:1347] Generated new cluster ID: 0091cb0ce1754befb7052dfb459d8ff4
I20260808 04:53:10.608775 4396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:10.617502 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00beb1e70f4a47679f825bd15c2e17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.617635 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:10.619822 4325 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00beb1e70f4a47679f825bd15c2e17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beb72a64d61c4b30ba6441e0d8dd0f85" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43601 } } peers { permanent_uuid: "00beb1e70f4a47679f825bd15c2e17df" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 43031 } } peers { permanent_uuid: "bbf22b09a8104149a8c6a493a48bb631" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 45423 } } }
I20260808 04:53:10.619946 4325 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:53:10.630967 4396 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:53:10.632179 4396 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:10.659252 4396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00beb1e70f4a47679f825bd15c2e17df: Generated new TSK 0
I20260808 04:53:10.660094 4396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:53:11.503926 4346 catalog_manager.cc:1259] Loaded cluster ID: 0091cb0ce1754befb7052dfb459d8ff4
I20260808 04:53:11.504019 4346 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: loading cluster ID for follower catalog manager: success
I20260808 04:53:11.505678 4346 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: acquiring CA information for follower catalog manager: success
I20260808 04:53:11.506919 4346 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbf22b09a8104149a8c6a493a48bb631: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:53:11.525480 4370 catalog_manager.cc:1259] Loaded cluster ID: 0091cb0ce1754befb7052dfb459d8ff4
I20260808 04:53:11.525549 4370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: loading cluster ID for follower catalog manager: success
I20260808 04:53:11.527163 4370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: acquiring CA information for follower catalog manager: success
I20260808 04:53:11.528312 4370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P beb72a64d61c4b30ba6441e0d8dd0f85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:53:11.745424 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 3981
I20260808 04:53:11.761036 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 4088
I20260808 04:53:11.772010 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 4197
2026-08-08T04:53:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3356 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:53:11 dist-test-slave-zpvx krb5kdc[4427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:53:11 dist-test-slave-zpvx krb5kdc[4427](info): set up 2 sockets
Aug 08 04:53:11 dist-test-slave-zpvx krb5kdc[4427](info): commencing operation
krb5kdc: starting...
W20260808 04:53:13.045953 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:53:13 dist-test-slave-zpvx krb5kdc[4427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:53:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:53:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:53:13.169523 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44365
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:13.280064 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:13.280306 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:13.280370 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:13.284226 4451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:13.284276 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:13.284294 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:13.284350 4451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:13.284409 4451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:13.289412 4451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:44365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:13.290390 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:13.291378 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:13.299715 4457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:13.299813 4451 server_base.cc:1034] running on GCE node
W20260808 04:53:13.299723 4460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:13.299749 4458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:13.300539 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:13.301599 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:13.302793 4451 hybrid_clock.cc:648] HybridClock initialized: now 1786164793302768 us; error 38 us; skew 500 ppm
Aug 08 04:53:13 dist-test-slave-zpvx krb5kdc[4427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164793, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:13.311044 4451 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:13.312281 4451 webserver.cc:460] Webserver started at http://127.15.30.126:42361/ using document root <none> and password file <none>
I20260808 04:53:13.312897 4451 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:13.312994 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:13.313205 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:13.315249 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "10ae5ca2a6024e8999e2f07980f1fbc2"
format_stamp: "Formatted at 2026-08-08 04:53:13 on dist-test-slave-zpvx"
server_key: "1591d6d22120f217c073cdbae4565b43"
server_key_iv: "c120fb6c769e0a43fedb8ca4057522e6"
server_key_version: "encryptionkey@0"
I20260808 04:53:13.315829 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "10ae5ca2a6024e8999e2f07980f1fbc2"
format_stamp: "Formatted at 2026-08-08 04:53:13 on dist-test-slave-zpvx"
server_key: "1591d6d22120f217c073cdbae4565b43"
server_key_iv: "c120fb6c769e0a43fedb8ca4057522e6"
server_key_version: "encryptionkey@0"
I20260808 04:53:13.319458 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:53:13.321826 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:13.323019 4451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 04:53:13.323186 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "10ae5ca2a6024e8999e2f07980f1fbc2"
format_stamp: "Formatted at 2026-08-08 04:53:13 on dist-test-slave-zpvx"
server_key: "1591d6d22120f217c073cdbae4565b43"
server_key_iv: "c120fb6c769e0a43fedb8ca4057522e6"
server_key_version: "encryptionkey@0"
I20260808 04:53:13.323303 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:13.347673 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:13.351004 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:13.351217 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:13.358426 4451 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:44365
I20260808 04:53:13.358455 4549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:44365 every 8 connection(s)
I20260808 04:53:13.359338 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:13.362150 4551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:13.365190 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 4451
I20260808 04:53:13.365336 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:53:13.365666 15481 external_mini_cluster.cc:1428] Setting key 3fbbfcf80b0ad83dea59e790ce7c7169
I20260808 04:53:13.368285 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2: Bootstrap starting.
I20260808 04:53:13.370467 4551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:13.371229 4551 log.cc:824] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:13.373173 4551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2: No bootstrap required, opened a new log
Aug 08 04:53:13 dist-test-slave-zpvx krb5kdc[4427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:13.377025 4551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44365 } }
I20260808 04:53:13.377310 4551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:13.377378 4551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ae5ca2a6024e8999e2f07980f1fbc2, State: Initialized, Role: FOLLOWER
I20260808 04:53:13.377883 4551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44365 } }
I20260808 04:53:13.378062 4551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:13.378170 4551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:13.378269 4551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:13.379354 4551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44365 } }
I20260808 04:53:13.379674 4551 leader_election.cc:302] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10ae5ca2a6024e8999e2f07980f1fbc2; no voters:
I20260808 04:53:13.379959 4551 leader_election.cc:288] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:53:13.380067 4556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:13.380214 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:13.367110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.15.30.126:44365)
0808 04:53:13.367574 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:13.367589 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:13.367625 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:53:13.368251 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:13.368268 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:13.368672 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:13.369011 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:13.369835 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:13.370903 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:13.371821 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:13.372282 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:13.372479 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:13.375216 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:13.375222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:13.375235 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:13.375261 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:13.377068 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:13.377563 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:13.377568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:13.377572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:13.377684 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:13.377981 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:13.377983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:13.377984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:13.378522 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:13.378686 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:13.378977 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 04:53:13.379265 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260808 04:53:13.380324 4556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 10ae5ca2a6024e8999e2f07980f1fbc2, State: Running, Role: LEADER
I20260808 04:53:13.380632 4556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44365 } }
I20260808 04:53:13.381240 4551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:13.381732 4558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44365 } } }
I20260808 04:53:13.381743 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10ae5ca2a6024e8999e2f07980f1fbc2. Latest consensus state: current_term: 1 leader_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae5ca2a6024e8999e2f07980f1fbc2" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44365 } } }
I20260808 04:53:13.381850 4558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:13.381868 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:13.382194 4561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:13.383384 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:53:13.385396 4561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:13.390869 4561 catalog_manager.cc:1347] Generated new cluster ID: bd794ac006cb4d1c88a7c571363c0988
I20260808 04:53:13.390933 4561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:13.405886 4561 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:53:13.407051 4561 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:13.413897 4561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10ae5ca2a6024e8999e2f07980f1fbc2: Generated new TSK 0
I20260808 04:53:13.414546 4561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:53:13.439298 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 4451
2026-08-08T04:53:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1668 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:53:13 dist-test-slave-zpvx krb5kdc[4593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:53:13 dist-test-slave-zpvx krb5kdc[4593](info): set up 2 sockets
Aug 08 04:53:13 dist-test-slave-zpvx krb5kdc[4593](info): commencing operation
krb5kdc: starting...
W20260808 04:53:14.690833 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:53:14 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:53:14.811478 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42771
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:42771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:14.922066 4621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:14.922334 4621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:14.922420 4621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:14.926357 4621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:14.926432 4621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:14.926469 4621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:14.926524 4621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:14.926563 4621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:14.932021 4621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:42771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:14.933110 4621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:14.934134 4621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:14.942436 4631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:14.942510 4633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:14.942430 4630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:14.942780 4621 server_base.cc:1034] running on GCE node
I20260808 04:53:14.943331 4621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:14.944411 4621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:14.945632 4621 hybrid_clock.cc:648] HybridClock initialized: now 1786164794945616 us; error 35 us; skew 500 ppm
Aug 08 04:53:14 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164794, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:14.948355 4621 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:14.949259 4621 webserver.cc:460] Webserver started at http://127.15.30.126:37283/ using document root <none> and password file <none>
I20260808 04:53:14.949810 4621 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:14.949899 4621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:14.950129 4621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:14.952085 4621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "1b06e201285443e78c2690872b8663ed"
format_stamp: "Formatted at 2026-08-08 04:53:14 on dist-test-slave-zpvx"
server_key: "f5310e19b62e307db4803b060acdb733"
server_key_iv: "b9b10d1c8da7c0d43b904419d227254a"
server_key_version: "encryptionkey@0"
I20260808 04:53:14.952690 4621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "1b06e201285443e78c2690872b8663ed"
format_stamp: "Formatted at 2026-08-08 04:53:14 on dist-test-slave-zpvx"
server_key: "f5310e19b62e307db4803b060acdb733"
server_key_iv: "b9b10d1c8da7c0d43b904419d227254a"
server_key_version: "encryptionkey@0"
I20260808 04:53:14.956162 4621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:53:14.958518 4642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:14.959697 4621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:14.959837 4621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "1b06e201285443e78c2690872b8663ed"
format_stamp: "Formatted at 2026-08-08 04:53:14 on dist-test-slave-zpvx"
server_key: "f5310e19b62e307db4803b060acdb733"
server_key_iv: "b9b10d1c8da7c0d43b904419d227254a"
server_key_version: "encryptionkey@0"
I20260808 04:53:14.959945 4621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:14.973884 4621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:14.976471 4621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:14.976686 4621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:14.983475 4621 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:42771
I20260808 04:53:14.983759 4724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:42771 every 8 connection(s)
I20260808 04:53:14.984422 4621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:14.986979 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 4621
I20260808 04:53:14.987033 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:14.987167 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:53:14.987459 15481 external_mini_cluster.cc:1428] Setting key df1b24339c041a579eaa112c20e79d19
I20260808 04:53:14.994107 4726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Bootstrap starting.
I20260808 04:53:14.996833 4726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Neither blocks nor log segments found. Creating new log.
Aug 08 04:53:14 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:14.997639 4726 log.cc:824] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:14.999814 4726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: No bootstrap required, opened a new log
I20260808 04:53:15.002249 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:14.988851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.15.30.126:42771)
0808 04:53:14.989310 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:14.989333 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:14.989376 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:53:14.990411 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:14.990437 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:14.990842 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:14.991205 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:14.991882 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:14.993393 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:14.994274 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:14.994741 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:14.994860 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:14.997665 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:14.997673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:14.997687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:14.997720 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:14.999463 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:14.999984 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:14.999989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:14.999993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.000108 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.000445 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.000448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.000449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.000806 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:15.000953 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.001280 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.001586 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260808 04:53:15.003926 4726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:15.004132 4726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:15.004194 4726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b06e201285443e78c2690872b8663ed, State: Initialized, Role: FOLLOWER
I20260808 04:53:15.004719 4726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:15.004843 4726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:15.004930 4726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:15.005046 4726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:15.005903 4726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:15.006250 4726 leader_election.cc:302] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b06e201285443e78c2690872b8663ed; no voters:
I20260808 04:53:15.006557 4726 leader_election.cc:288] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:53:15.006690 4734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:15.006937 4734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 LEADER]: Becoming Leader. State: Replica: 1b06e201285443e78c2690872b8663ed, State: Running, Role: LEADER
I20260808 04:53:15.007241 4734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:15.007764 4726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:15.008970 4737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b06e201285443e78c2690872b8663ed. Latest consensus state: current_term: 1 leader_uuid: "1b06e201285443e78c2690872b8663ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } } }
I20260808 04:53:15.009135 4737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:15.009442 4736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b06e201285443e78c2690872b8663ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } } }
I20260808 04:53:15.009523 4747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:15.009558 4736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:15.013065 4747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:15.018571 4747 catalog_manager.cc:1347] Generated new cluster ID: 2e3b32adcd3d44228b107b108f27f0a0
I20260808 04:53:15.018646 4747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:15.039646 4747 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:53:15.040832 4747 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:15.054564 4747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Generated new TSK 0
I20260808 04:53:15.055387 4747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:53:15.088227 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42771
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:15.201072 4769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:15.201305 4769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:15.201346 4769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:15.205286 4769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:15.205335 4769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:15.205391 4769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:53:15.210464 4769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:15.211369 4769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:15.212301 4769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:15.221851 4776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:15.221982 4769 server_base.cc:1034] running on GCE node
W20260808 04:53:15.221915 4778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:15.221877 4775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:15.222513 4769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:15.223125 4769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:15.224336 4769 hybrid_clock.cc:648] HybridClock initialized: now 1786164795224327 us; error 30 us; skew 500 ppm
Aug 08 04:53:15 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164795, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:15.227123 4769 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:53:15.227874 4769 webserver.cc:460] Webserver started at http://127.15.30.65:46117/ using document root <none> and password file <none>
I20260808 04:53:15.228463 4769 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:15.228530 4769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:15.228740 4769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:15.230676 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "3d0468efd23240ec9ed81bb8dc8ef345"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "398d4dc7407ff7169ab8c911651965f7"
server_key_iv: "dcb646002d2c6d16d8f990c3b5bdc77e"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.231238 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "3d0468efd23240ec9ed81bb8dc8ef345"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "398d4dc7407ff7169ab8c911651965f7"
server_key_iv: "dcb646002d2c6d16d8f990c3b5bdc77e"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.235137 4769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:53:15.237517 4787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.238700 4769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:53:15.238844 4769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "3d0468efd23240ec9ed81bb8dc8ef345"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "398d4dc7407ff7169ab8c911651965f7"
server_key_iv: "dcb646002d2c6d16d8f990c3b5bdc77e"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.238950 4769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:15.259895 4769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:15.262691 4769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:15.262918 4769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:15.263505 4769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:15.264487 4769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:15.264564 4769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.264631 4769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:15.264683 4769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.273995 4769 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:46303
I20260808 04:53:15.274014 4970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:46303 every 8 connection(s)
I20260808 04:53:15.274932 4769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
Aug 08 04:53:15 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164795, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:15.284773 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 4769
I20260808 04:53:15.284899 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:53:15.285281 15481 external_mini_cluster.cc:1428] Setting key 13a767ed6a55dd3cb092e33b4f334fdd
I20260808 04:53:15.289080 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.276785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:40343 (local address 127.15.30.126:42771)
0808 04:53:15.276920 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.276924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.277747 (+ 823us) server_negotiation.cc:407] Connection header received
0808 04:53:15.279150 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.279154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.279234 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.279330 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:15.280528 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.281064 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.282204 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.282436 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.282549 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.284975 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:15.284979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:15.284984 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:15.285015 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:15.286610 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.287269 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.287275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.287278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.287378 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.287741 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.287747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.287751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.288009 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:15.288120 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.288661 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.288814 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:53:15.289767 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.277154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:40343)
0808 04:53:15.277597 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.277638 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.278924 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.279508 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.279514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.279873 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:15.280332 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.280346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.281190 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.281195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.282066 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.282076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.282590 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.282594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.282769 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.283290 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:15.283305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:15.284768 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:15.286787 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:15.286794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:15.286809 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:15.287147 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:15.287488 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:15.287490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:15.287491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:15.287624 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:15.288122 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:15.288127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:15.288350 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.288580 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.288844 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260808 04:53:15.291024 4971 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42771
I20260808 04:53:15.291288 4971 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:15.291769 4971 heartbeater.cc:511] Master 127.15.30.126:42771 requested a full tablet report, sending...
I20260808 04:53:15.293332 4668 ts_manager.cc:194] Registered new tserver with Master: 3d0468efd23240ec9ed81bb8dc8ef345 (127.15.30.65:46303)
I20260808 04:53:15.294436 4668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:40343
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:53:15.314801 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42771
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:15.425547 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:15.425828 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:15.425905 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:15.429863 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:15.429948 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:15.430039 4977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:53:15.435070 4977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:15.436074 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:15.437110 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:15.446633 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:15.446725 4977 server_base.cc:1034] running on GCE node
W20260808 04:53:15.446679 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:15.446686 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:15.447263 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:15.447881 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:15.449210 4977 hybrid_clock.cc:648] HybridClock initialized: now 1786164795449186 us; error 32 us; skew 500 ppm
Aug 08 04:53:15 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164795, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:15.452126 4977 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:53:15.452984 4977 webserver.cc:460] Webserver started at http://127.15.30.66:40303/ using document root <none> and password file <none>
I20260808 04:53:15.453487 4977 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:15.453552 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:15.453709 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:15.455641 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "987742573cea4391a897720d3a1d263a"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "80f6c5e8b3da48843b0f831d4301c991"
server_key_iv: "9d9db61dfefad29c6e25a2f56ef27b16"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.456169 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "987742573cea4391a897720d3a1d263a"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "80f6c5e8b3da48843b0f831d4301c991"
server_key_iv: "9d9db61dfefad29c6e25a2f56ef27b16"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.459832 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:53:15.462210 4998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.463425 4977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:15.463570 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "987742573cea4391a897720d3a1d263a"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "80f6c5e8b3da48843b0f831d4301c991"
server_key_iv: "9d9db61dfefad29c6e25a2f56ef27b16"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.463671 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:15.488353 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:15.491638 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:15.491793 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:15.492321 4977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:15.493105 4977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:15.493151 4977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.493199 4977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:15.493216 4977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.502301 4977 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:41817
I20260808 04:53:15.502322 5180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:41817 every 8 connection(s)
I20260808 04:53:15.503153 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:53:15.511392 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 4977
I20260808 04:53:15.511533 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:53:15.511895 15481 external_mini_cluster.cc:1428] Setting key aadcefc299f062ae1125a937692be3bb
Aug 08 04:53:15 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164795, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:15.518150 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.505140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:32901 (local address 127.15.30.126:42771)
0808 04:53:15.505279 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.505283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.506150 (+ 867us) server_negotiation.cc:407] Connection header received
0808 04:53:15.507574 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.507577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.507638 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.507755 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:15.509089 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.509631 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.510805 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.511049 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.511186 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.514279 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:15.514282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:15.514284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:15.514308 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:15.515770 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.516459 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.516461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.516462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.516541 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.516966 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.516971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.516973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.517202 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:15.517283 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.517936 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.518036 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:53:15.519393 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.505508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:32901)
0808 04:53:15.505999 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.506041 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.507362 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.507952 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.507958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.508316 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:15.508862 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.508875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.509752 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.509754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.510679 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.510690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.511277 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.511284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.511491 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.512216 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:15.512232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:15.514078 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:15.515890 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:15.515897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:15.515912 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:15.516286 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:15.516639 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:15.516644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:15.516646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:15.516845 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:15.517347 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:15.517356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:15.517683 (+ 327us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.517899 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.518250 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260808 04:53:15.520659 5181 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42771
I20260808 04:53:15.520911 5181 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:15.521387 5181 heartbeater.cc:511] Master 127.15.30.126:42771 requested a full tablet report, sending...
I20260808 04:53:15.522478 4668 ts_manager.cc:194] Registered new tserver with Master: 987742573cea4391a897720d3a1d263a (127.15.30.66:41817)
I20260808 04:53:15.523033 4668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:32901
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:53:15.541615 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42771
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:15.654554 5191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:15.654778 5191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:15.654817 5191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:15.658736 5191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:15.658787 5191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:15.658867 5191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:53:15.663887 5191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:15.664873 5191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:15.665906 5191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:15.675805 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:15.675894 5191 server_base.cc:1034] running on GCE node
W20260808 04:53:15.675832 5204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:15.675832 5201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:15.676538 5191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:15.677198 5191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:15.678375 5191 hybrid_clock.cc:648] HybridClock initialized: now 1786164795678357 us; error 36 us; skew 500 ppm
Aug 08 04:53:15 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164795, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:15.681262 5191 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:53:15.682211 5191 webserver.cc:460] Webserver started at http://127.15.30.67:41947/ using document root <none> and password file <none>
I20260808 04:53:15.682818 5191 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:15.682885 5191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:15.683177 5191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:15.685521 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "12b521e0d9f34a3dad0b1b50dacbb1be"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "bd86b7caba91979bbeea40d188debb9b"
server_key_iv: "3eff05d45476ff83024f0f4c17682d26"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.686132 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "12b521e0d9f34a3dad0b1b50dacbb1be"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "bd86b7caba91979bbeea40d188debb9b"
server_key_iv: "3eff05d45476ff83024f0f4c17682d26"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.690521 5191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:53:15.692975 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.694269 5191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:53:15.694442 5191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "12b521e0d9f34a3dad0b1b50dacbb1be"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "bd86b7caba91979bbeea40d188debb9b"
server_key_iv: "3eff05d45476ff83024f0f4c17682d26"
server_key_version: "encryptionkey@0"
I20260808 04:53:15.694564 5191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:15.720690 5191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:15.724289 5191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:15.724536 5191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:15.725193 5191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:15.726069 5191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:15.726158 5191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.726230 5191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:15.726285 5191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:15.736001 5191 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:37833
I20260808 04:53:15.736029 5396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:37833 every 8 connection(s)
I20260808 04:53:15.736935 5191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:53:15.738075 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 5191
I20260808 04:53:15.738211 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:53:15.738567 15481 external_mini_cluster.cc:1428] Setting key 97ac9de090bbbdb194c06afba2f491b1
Aug 08 04:53:15 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164795, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:15.751497 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.739060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:38559 (local address 127.15.30.126:42771)
0808 04:53:15.739221 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.739226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.740274 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 04:53:15.741945 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.741947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.741992 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.742063 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:15.743349 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.743939 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.745220 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.745430 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.745510 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.748029 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:15.748032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:15.748034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:15.748059 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:15.749519 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.750025 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.750030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.750032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.750115 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.750456 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.750460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.750462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.750637 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:15.750748 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.751223 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.751340 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:53:15.752586 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.739528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:38559)
0808 04:53:15.740116 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.740172 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.741739 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.742246 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.742253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.742719 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:15.743211 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.743225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.744054 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.744057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.745101 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.745112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.745567 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.745570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.745737 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.746257 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:15.746274 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:15.747859 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:15.749642 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:15.749648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:15.749661 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:15.749917 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:15.750216 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:15.750219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:15.750220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:15.750357 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:15.750758 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:15.750766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:15.750993 (+ 227us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.751178 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.751501 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":157,"threads_started":1}
I20260808 04:53:15.753674 5397 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42771
I20260808 04:53:15.753876 5397 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:15.754351 5397 heartbeater.cc:511] Master 127.15.30.126:42771 requested a full tablet report, sending...
I20260808 04:53:15.755311 4668 ts_manager.cc:194] Registered new tserver with Master: 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
I20260808 04:53:15.755884 4668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:38559
I20260808 04:53:15.765065 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:53:15.775352 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.767273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.15.30.126:42771)
0808 04:53:15.767435 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.767442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.767646 (+ 204us) server_negotiation.cc:407] Connection header received
0808 04:53:15.767816 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.767819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.767874 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.767979 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:15.768777 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.769371 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.770198 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.770411 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.770494 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.771908 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:15.771911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:15.771913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:15.771936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:15.773626 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.774077 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.774082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.774083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.774168 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:15.774447 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:15.774450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:15.774452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:15.774669 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:15.774818 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.775059 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.775189 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 04:53:15.778301 4668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:53:15.781280 4668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:53:15.795449 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.789590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37833 (local address 127.0.0.1:54660)
0808 04:53:15.790231 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.790278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.790431 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.791037 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.791044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.791073 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:15.791215 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.791227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.792551 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.792556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.793655 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.793666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.794888 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.794892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.794998 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.795037 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.795156 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.795285 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":66,"threads_started":1}
I20260808 04:53:15.795486 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.789722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.15.30.67:37833)
0808 04:53:15.790101 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.790113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.790373 (+ 260us) server_negotiation.cc:407] Connection header received
0808 04:53:15.790634 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.790641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.790872 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.791046 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:15.791360 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.792433 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.793922 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.794762 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.794869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.795007 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.795219 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.795319 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":158,"threads_started":1}
I20260808 04:53:15.795837 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.789671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46303 (local address 127.0.0.1:47566)
0808 04:53:15.790543 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.790557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.790634 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.791527 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.791531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.791549 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:15.791646 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.791652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.793186 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.793190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.794341 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.794352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.795422 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.795426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.795536 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.795566 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.795625 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.795697 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":46,"thread_start_us":127,"threads_started":1}
I20260808 04:53:15.796001 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.790312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41817 (local address 127.0.0.1:34886)
0808 04:53:15.791307 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.791320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.791408 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.791940 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.791942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.791963 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:15.792053 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.792059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.793398 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.793403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.794417 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.794429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.795598 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.795604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.795701 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.795723 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.795794 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.795869 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260808 04:53:15.796125 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.790486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.15.30.66:41817)
0808 04:53:15.790902 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.790915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.791512 (+ 597us) server_negotiation.cc:407] Connection header received
0808 04:53:15.791590 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.791595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.791791 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.791937 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:15.792172 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.793275 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.794525 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.795290 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.795390 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.795525 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.795857 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.795972 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":96,"threads_started":1}
I20260808 04:53:15.796728 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.790753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.15.30.65:46303)
0808 04:53:15.791046 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.791052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.791074 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:53:15.791150 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.791155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.791386 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.791782 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:15.791914 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.793060 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.794485 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.795283 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.796317 (+ 1034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.796489 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.796539 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.796614 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260808 04:53:15.798902 5296 tablet_service.cc:1467] Processing CreateTablet for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 (DEFAULT_TABLE table=test-table [id=21aa23e62b98460abb8678cd3b585e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:15.799155 5079 tablet_service.cc:1467] Processing CreateTablet for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 (DEFAULT_TABLE table=test-table [id=21aa23e62b98460abb8678cd3b585e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:15.799321 4866 tablet_service.cc:1467] Processing CreateTablet for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 (DEFAULT_TABLE table=test-table [id=21aa23e62b98460abb8678cd3b585e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:15.799971 5296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:15.800192 5079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:15.800199 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:15.805857 5422 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Bootstrap starting.
I20260808 04:53:15.807036 5423 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap starting.
I20260808 04:53:15.809002 5424 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Bootstrap starting.
I20260808 04:53:15.809212 5422 tablet_bootstrap.cc:654] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:15.809388 5423 tablet_bootstrap.cc:654] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:15.810195 5423 log.cc:824] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:15.810195 5422 log.cc:824] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:15.811429 5424 tablet_bootstrap.cc:654] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:15.812120 5424 log.cc:824] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:15.812274 5422 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: No bootstrap required, opened a new log
I20260808 04:53:15.812300 5423 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: No bootstrap required, opened a new log
I20260808 04:53:15.812503 5422 ts_tablet_manager.cc:1397] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:53:15.812503 5423 ts_tablet_manager.cc:1397] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:15.814043 5424 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: No bootstrap required, opened a new log
I20260808 04:53:15.814249 5424 ts_tablet_manager.cc:1397] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:53:15.816576 5422 raft_consensus.cc:348] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.816859 5422 raft_consensus.cc:374] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:15.816954 5422 raft_consensus.cc:729] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b521e0d9f34a3dad0b1b50dacbb1be, State: Initialized, Role: FOLLOWER
I20260808 04:53:15.816977 5423 raft_consensus.cc:348] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.817028 5424 raft_consensus.cc:348] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.817305 5424 raft_consensus.cc:374] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:15.817312 5423 raft_consensus.cc:374] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:15.817369 5423 raft_consensus.cc:729] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0468efd23240ec9ed81bb8dc8ef345, State: Initialized, Role: FOLLOWER
I20260808 04:53:15.817368 5424 raft_consensus.cc:729] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987742573cea4391a897720d3a1d263a, State: Initialized, Role: FOLLOWER
I20260808 04:53:15.817426 5422 consensus_queue.cc:260] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.817832 5424 consensus_queue.cc:260] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.817826 5423 consensus_queue.cc:260] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.818420 5422 ts_tablet_manager.cc:1428] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:15.818775 5423 ts_tablet_manager.cc:1428] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260808 04:53:15.818805 5424 ts_tablet_manager.cc:1428] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260808 04:53:15.819020 5397 heartbeater.cc:503] Master 127.15.30.126:42771 was elected leader, sending a full tablet report...
I20260808 04:53:15.819193 5181 heartbeater.cc:503] Master 127.15.30.126:42771 was elected leader, sending a full tablet report...
I20260808 04:53:15.819213 4971 heartbeater.cc:503] Master 127.15.30.126:42771 was elected leader, sending a full tablet report...
I20260808 04:53:15.922181 5439 raft_consensus.cc:482] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:15.922430 5439 raft_consensus.cc:504] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.923673 5439 leader_election.cc:288] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833), 987742573cea4391a897720d3a1d263a (127.15.30.66:41817)
I20260808 04:53:15.927817 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.924078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:58257 (local address 127.15.30.66:41817)
0808 04:53:15.924220 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.924226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.924557 (+ 331us) server_negotiation.cc:407] Connection header received
0808 04:53:15.924735 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.924738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.924785 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.924876 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:15.925206 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.925733 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.926840 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.927240 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.927372 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.927427 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.927605 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.927675 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:53:15.927841 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.924066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:55007 (local address 127.15.30.67:37833)
0808 04:53:15.924222 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.924227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.924544 (+ 317us) server_negotiation.cc:407] Connection header received
0808 04:53:15.924740 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.924744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.924793 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.924880 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:15.925209 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.925707 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.926862 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.927265 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.927395 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.927450 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.927649 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.927720 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:53:15.927846 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.923958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41817 (local address 127.15.30.65:58257)
0808 04:53:15.924439 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.924452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.924598 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.924888 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.924891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.924926 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:15.925082 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.925087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.925853 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.925856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.926717 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.926726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.927368 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.927372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.927467 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.927500 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.927586 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.927652 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":209,"spinlock_wait_cycles":18688,"thread_start_us":82,"threads_started":1}
I20260808 04:53:15.927862 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.923958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37833 (local address 127.15.30.65:55007)
0808 04:53:15.924441 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:15.924455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:15.924624 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:15.924893 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:15.924896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:15.924930 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:15.925094 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.925101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.925826 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.925830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.926723 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:15.926733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.927394 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.927397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:15.927508 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.927529 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:53:15.927634 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:53:15.927702 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":69,"spinlock_wait_cycles":2304,"thread_start_us":171,"threads_started":1}
I20260808 04:53:15.928643 5329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56a02ed42f34fa8aea7ef79ac0ef4f7" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" is_pre_election: true
I20260808 04:53:15.928644 5111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56a02ed42f34fa8aea7ef79ac0ef4f7" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987742573cea4391a897720d3a1d263a" is_pre_election: true
I20260808 04:53:15.928941 5329 raft_consensus.cc:2454] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 0.
I20260808 04:53:15.928961 5111 raft_consensus.cc:2454] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 0.
I20260808 04:53:15.929427 4790 leader_election.cc:302] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 3d0468efd23240ec9ed81bb8dc8ef345; no voters:
I20260808 04:53:15.929693 5439 raft_consensus.cc:2781] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:53:15.929780 5439 raft_consensus.cc:482] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:53:15.929811 5439 raft_consensus.cc:3037] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:15.930878 5439 raft_consensus.cc:504] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.931253 5439 leader_election.cc:288] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 election: Requested vote from peers 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833), 987742573cea4391a897720d3a1d263a (127.15.30.66:41817)
I20260808 04:53:15.931653 5111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56a02ed42f34fa8aea7ef79ac0ef4f7" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987742573cea4391a897720d3a1d263a"
I20260808 04:53:15.931641 5329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56a02ed42f34fa8aea7ef79ac0ef4f7" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be"
I20260808 04:53:15.931767 5329 raft_consensus.cc:3037] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:15.931768 5111 raft_consensus.cc:3037] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:15.933086 5111 raft_consensus.cc:2454] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 1.
I20260808 04:53:15.933086 5329 raft_consensus.cc:2454] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 1.
I20260808 04:53:15.933382 4790 leader_election.cc:302] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 3d0468efd23240ec9ed81bb8dc8ef345, 987742573cea4391a897720d3a1d263a; no voters:
I20260808 04:53:15.933604 5439 raft_consensus.cc:2781] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:15.933841 5439 raft_consensus.cc:686] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0468efd23240ec9ed81bb8dc8ef345, State: Running, Role: LEADER
I20260808 04:53:15.934167 5439 consensus_queue.cc:237] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:15.937333 4667 catalog_manager.cc:5495] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0468efd23240ec9ed81bb8dc8ef345 (127.15.30.65). New cstate: current_term: 1 leader_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } health_report { overall_health: UNKNOWN } } }
I20260808 04:53:15.953130 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:15.949591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.15.30.65:46303)
0808 04:53:15.949737 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:15.949742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:15.949756 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:53:15.949922 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:15.949925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:15.949975 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:15.950081 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:53:15.950351 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.950852 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.951700 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:15.951923 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:15.952033 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:15.952348 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:53:15.952745 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:53:15.952876 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:15.952942 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 04:53:15.952995 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:53:15.958721 5111 raft_consensus.cc:1261] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Refusing update from remote peer 3d0468efd23240ec9ed81bb8dc8ef345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:15.958722 5329 raft_consensus.cc:1261] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Refusing update from remote peer 3d0468efd23240ec9ed81bb8dc8ef345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:15.959424 5442 consensus_queue.cc:1035] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:15.959571 5439 consensus_queue.cc:1035] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:15.968309 5449 mvcc.cc:205] Tried to move back new op lower bound from 7316131004240375808 to 7316131004149084160. Current Snapshot: MvccSnapshot[applied={T|T < 7316131004240375808}]
I20260808 04:53:15.975816 4667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:15.975951 4667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:15.978197 4667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Sending DeleteTablet for 3 replicas of tablet e56a02ed42f34fa8aea7ef79ac0ef4f7
I20260808 04:53:15.978904 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:15 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:54660
I20260808 04:53:15.979015 5079 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:15 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:34886
I20260808 04:53:15.979393 5459 tablet_replica.cc:330] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: stopping tablet replica
I20260808 04:53:15.979398 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:15 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:47566
I20260808 04:53:15.979621 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 4769
I20260808 04:53:15.979759 5459 raft_consensus.cc:2229] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:15.982349 5460 tablet_replica.cc:330] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: stopping tablet replica
I20260808 04:53:15.982590 5460 raft_consensus.cc:2229] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:15.982832 5460 raft_consensus.cc:2258] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:15.992281 5459 raft_consensus.cc:2258] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:15.993250 5460 ts_tablet_manager.cc:1905] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:15.994772 5459 ts_tablet_manager.cc:1905] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:15.996273 5460 ts_tablet_manager.cc:1918] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:15.996486 5460 log.cc:1196] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/e56a02ed42f34fa8aea7ef79ac0ef4f7
I20260808 04:53:15.996904 5460 ts_tablet_manager.cc:1939] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 987742573cea4391a897720d3a1d263a: Deleting consensus metadata
I20260808 04:53:15.997614 5459 ts_tablet_manager.cc:1918] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:15.997779 5459 log.cc:1196] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/e56a02ed42f34fa8aea7ef79ac0ef4f7
I20260808 04:53:15.998171 5459 ts_tablet_manager.cc:1939] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 12b521e0d9f34a3dad0b1b50dacbb1be: Deleting consensus metadata
I20260808 04:53:15.998353 4651 catalog_manager.cc:4841] TS 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 (table test-table [id=21aa23e62b98460abb8678cd3b585e4a]) successfully deleted
I20260808 04:53:15.999352 4647 catalog_manager.cc:4841] TS 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 (table test-table [id=21aa23e62b98460abb8678cd3b585e4a]) successfully deleted
W20260808 04:53:15.999389 4651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:46303 (error 108)
I20260808 04:53:15.999722 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 4977
I20260808 04:53:16.018304 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 5191
I20260808 04:53:16.031399 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.018561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46303 (local address 127.0.0.1:47590)
0808 04:53:16.031186 (+ 12625us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:16.031277 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12497}
W20260808 04:53:16.033443 4651 catalog_manager.cc:4584] TS 3d0468efd23240ec9ed81bb8dc8ef345 (127.15.30.65:46303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46303: connect: Connection refused (error 111)
I20260808 04:53:16.038305 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 4621
I20260808 04:53:16.051282 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42771
--webserver_interface=127.15.30.126
--webserver_port=37283
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:42771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:16.162942 5465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:16.163246 5465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:16.163342 5465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:16.167500 5465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:16.167572 5465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:16.167606 5465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:16.167662 5465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:16.167716 5465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:16.173056 5465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:42771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:16.174042 5465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:16.175019 5465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:16.190330 5485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:16.190339 5481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:16.190579 5465 server_base.cc:1034] running on GCE node
W20260808 04:53:16.190341 5477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:16.191236 5465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:16.192327 5465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:16.193538 5465 hybrid_clock.cc:648] HybridClock initialized: now 1786164796193526 us; error 25 us; skew 500 ppm
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:16.196352 5465 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:16.197284 5465 webserver.cc:460] Webserver started at http://127.15.30.126:37283/ using document root <none> and password file <none>
I20260808 04:53:16.197863 5465 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:16.197970 5465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:16.201797 5465 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 04:53:16.203965 5495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.205227 5465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:53:16.205379 5465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "1b06e201285443e78c2690872b8663ed"
format_stamp: "Formatted at 2026-08-08 04:53:14 on dist-test-slave-zpvx"
server_key: "f5310e19b62e307db4803b060acdb733"
server_key_iv: "b9b10d1c8da7c0d43b904419d227254a"
server_key_version: "encryptionkey@0"
I20260808 04:53:16.205963 5465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:16.215996 5465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:16.219159 5465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:16.219379 5465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:16.226557 5465 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:42771
I20260808 04:53:16.226864 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:42771 every 8 connection(s)
I20260808 04:53:16.227630 5465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:16.228008 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 5465
I20260808 04:53:16.228554 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:46303
--local_ip_for_outbound_sockets=127.15.30.65
--tserver_master_addrs=127.15.30.126:42771
--webserver_port=46117
--webserver_interface=127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:16.232810 5582 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:53:16.234563 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Bootstrap starting.
I20260808 04:53:16.251675 5582 log.cc:824] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:16.261348 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:53:16.261735 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Bootstrap complete.
I20260808 04:53:16.266125 5582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:16.266574 5582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b06e201285443e78c2690872b8663ed, State: Initialized, Role: FOLLOWER
I20260808 04:53:16.267071 5582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:16.267215 5582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:16.267313 5582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:16.267437 5582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:53:16.269390 5582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:16.269735 5582 leader_election.cc:302] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b06e201285443e78c2690872b8663ed; no voters:
I20260808 04:53:16.270102 5582 leader_election.cc:288] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:53:16.270195 5586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:53:16.270411 5586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 LEADER]: Becoming Leader. State: Replica: 1b06e201285443e78c2690872b8663ed, State: Running, Role: LEADER
I20260808 04:53:16.270761 5586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:16.271334 5582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:16.271708 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b06e201285443e78c2690872b8663ed. Latest consensus state: current_term: 2 leader_uuid: "1b06e201285443e78c2690872b8663ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } } }
I20260808 04:53:16.271924 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:16.272094 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b06e201285443e78c2690872b8663ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } } }
I20260808 04:53:16.272293 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:16.272414 5595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:16.277429 5595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:16.278059 5595 catalog_manager.cc:1259] Loaded cluster ID: 2e3b32adcd3d44228b107b108f27f0a0
I20260808 04:53:16.278124 5595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:16.279541 5595 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 04:53:16.284456 5595 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 04:53:16.289768 5595 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:16.290452 5595 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Loaded TSK: 0
I20260808 04:53:16.291596 5595 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:53:16.345485 5584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:16.345757 5584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:16.345830 5584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:16.349799 5584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:16.349877 5584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:16.349965 5584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:53:16.354964 5584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:46303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=46117
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:16.355919 5584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:16.356956 5584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:16.366575 5630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:16.366608 5628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:16.366734 5584 server_base.cc:1034] running on GCE node
W20260808 04:53:16.366593 5633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:16.367350 5584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:16.367969 5584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:16.369166 5584 hybrid_clock.cc:648] HybridClock initialized: now 1786164796369135 us; error 49 us; skew 500 ppm
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:16.372157 5584 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:53:16.373458 5584 webserver.cc:460] Webserver started at http://127.15.30.65:46117/ using document root <none> and password file <none>
I20260808 04:53:16.374142 5584 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:16.374248 5584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:16.379215 5584 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:53:16.381847 5641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.383356 5584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:53:16.383489 5584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "3d0468efd23240ec9ed81bb8dc8ef345"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "398d4dc7407ff7169ab8c911651965f7"
server_key_iv: "dcb646002d2c6d16d8f990c3b5bdc77e"
server_key_version: "encryptionkey@0"
I20260808 04:53:16.384181 5584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:16.407039 5584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:16.410593 5584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:16.410781 5584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:16.411442 5584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:16.412725 5655 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:53:16.416849 5584 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:53:16.416920 5584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 04:53:16.416981 5584 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:53:16.419147 5584 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:53:16.419204 5584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:53:16.419567 5655 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap starting.
I20260808 04:53:16.430140 5584 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:46303
I20260808 04:53:16.431191 5584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:53:16.431231 5819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:46303 every 8 connection(s)
I20260808 04:53:16.437023 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 5584
I20260808 04:53:16.437474 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:41817
--local_ip_for_outbound_sockets=127.15.30.66
--tserver_master_addrs=127.15.30.126:42771
--webserver_port=40303
--webserver_interface=127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:16.440351 5655 log.cc:824] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:16.448164 5655 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:53:16.448639 5655 tablet_bootstrap.cc:492] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap complete.
I20260808 04:53:16.450030 5655 ts_tablet_manager.cc:1397] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.014s
I20260808 04:53:16.452780 5655 raft_consensus.cc:348] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:16.453656 5655 raft_consensus.cc:729] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0468efd23240ec9ed81bb8dc8ef345, State: Initialized, Role: FOLLOWER
I20260808 04:53:16.453943 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.435137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:48869 (local address 127.15.30.126:42771)
0808 04:53:16.435709 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.435727 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.436205 (+ 478us) server_negotiation.cc:407] Connection header received
0808 04:53:16.438025 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.438042 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.438448 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.438799 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:16.440248 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.441706 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.443674 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.444341 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.444536 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.447843 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:16.447851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:16.447871 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:16.447912 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:16.450321 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.450960 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.450969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.450979 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.451148 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.451573 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.451576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.451578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.451998 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:16.452121 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.452789 (+ 668us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.453074 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":149,"threads_started":1}
I20260808 04:53:16.454200 5655 consensus_queue.cc:260] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } } peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } }
I20260808 04:53:16.454442 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.435074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:48869)
0808 04:53:16.436015 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:16.436071 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:16.437344 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:16.438906 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:16.438914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:16.439404 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:16.440043 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.440063 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.441905 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.441912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.443513 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.443530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.444512 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.444518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.444785 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.445520 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:16.445568 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:16.447617 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:16.450456 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.450462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.450474 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.450827 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.451277 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.451281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.451283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.451455 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.452140 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:16.452146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:16.452452 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 04:53:16.452652 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 04:53:16.452977 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":1,"thread_start_us":160,"threads_started":1}
I20260808 04:53:16.455201 5655 ts_tablet_manager.cc:1428] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 04:53:16.456736 5820 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42771
I20260808 04:53:16.456981 5820 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:16.457515 5820 heartbeater.cc:511] Master 127.15.30.126:42771 requested a full tablet report, sending...
I20260808 04:53:16.460022 5525 ts_manager.cc:194] Registered new tserver with Master: 3d0468efd23240ec9ed81bb8dc8ef345 (127.15.30.65:46303)
I20260808 04:53:16.463274 5525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:48869
I20260808 04:53:16.466050 5820 heartbeater.cc:503] Master 127.15.30.126:42771 was elected leader, sending a full tablet report...
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:53:16.475167 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.462762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46303 (local address 127.0.0.1:47602)
0808 04:53:16.463584 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:16.463624 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:16.463768 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:16.464271 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:16.464278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:16.464742 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:16.464877 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.464885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.467304 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.467308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.468089 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.468100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.468815 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.468819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.468937 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.469490 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:16.469502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:16.471013 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:16.473529 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.473537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.473542 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.473774 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.474096 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.474098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.474100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.474178 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.474667 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:16.474672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:16.474770 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 04:53:16.474878 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 04:53:16.475014 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":117,"threads_started":1}
I20260808 04:53:16.475517 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.462878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.15.30.65:46303)
0808 04:53:16.463470 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.463482 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.463726 (+ 244us) server_negotiation.cc:407] Connection header received
0808 04:53:16.463918 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.463923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.464116 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.464253 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:16.465011 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.467174 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.468211 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.468675 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.468811 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.471197 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:16.471208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:16.471221 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:16.471247 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:16.473417 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.473895 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.473901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.473903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.473999 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.474265 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.474268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.474270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.474557 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:16.474688 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.474985 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.475147 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":502,"thread_start_us":88,"threads_started":1}
I20260808 04:53:16.476447 5725 tablet_service.cc:1514] Processing DeleteTablet for tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:15 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:47602
I20260808 04:53:16.477487 5834 tablet_replica.cc:330] stopping tablet replica
I20260808 04:53:16.477775 5834 raft_consensus.cc:2229] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:16.478013 5834 raft_consensus.cc:2258] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:16.478935 5834 ts_tablet_manager.cc:1905] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:16.482378 5834 ts_tablet_manager.cc:1918] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:16.482496 5834 log.cc:1196] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/e56a02ed42f34fa8aea7ef79ac0ef4f7
I20260808 04:53:16.482786 5834 ts_tablet_manager.cc:1939] T e56a02ed42f34fa8aea7ef79ac0ef4f7 P 3d0468efd23240ec9ed81bb8dc8ef345: Deleting consensus metadata
I20260808 04:53:16.484457 5505 catalog_manager.cc:4841] TS 3d0468efd23240ec9ed81bb8dc8ef345 (127.15.30.65:46303): tablet e56a02ed42f34fa8aea7ef79ac0ef4f7 (table test-table [id=21aa23e62b98460abb8678cd3b585e4a]) successfully deleted
W20260808 04:53:16.557837 5828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:16.558079 5828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:16.558116 5828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:16.561879 5828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:16.561936 5828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:16.562008 5828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:53:16.567215 5828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:41817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=40303
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:16.568110 5828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:16.569106 5828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:16.578852 5842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:16.578977 5828 server_base.cc:1034] running on GCE node
W20260808 04:53:16.578867 5840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:16.578867 5839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:16.579536 5828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:16.580112 5828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:16.581269 5828 hybrid_clock.cc:648] HybridClock initialized: now 1786164796581229 us; error 52 us; skew 500 ppm
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:16.584064 5828 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:53:16.584929 5828 webserver.cc:460] Webserver started at http://127.15.30.66:40303/ using document root <none> and password file <none>
I20260808 04:53:16.585484 5828 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:16.585558 5828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:16.589293 5828 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:53:16.591300 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.592532 5828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:53:16.592657 5828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "987742573cea4391a897720d3a1d263a"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "80f6c5e8b3da48843b0f831d4301c991"
server_key_iv: "9d9db61dfefad29c6e25a2f56ef27b16"
server_key_version: "encryptionkey@0"
I20260808 04:53:16.593178 5828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:16.620671 5828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:16.623871 5828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:16.624078 5828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:16.624737 5828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:16.625554 5828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:16.625626 5828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.625691 5828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:16.625739 5828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.634617 5828 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:41817
I20260808 04:53:16.634655 6028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:41817 every 8 connection(s)
I20260808 04:53:16.635596 5828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:53:16.635959 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 5828
I20260808 04:53:16.636480 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:37833
--local_ip_for_outbound_sockets=127.15.30.67
--tserver_master_addrs=127.15.30.126:42771
--webserver_port=41947
--webserver_interface=127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:16.650739 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.637622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:50287 (local address 127.15.30.126:42771)
0808 04:53:16.637793 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.637798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.638726 (+ 928us) server_negotiation.cc:407] Connection header received
0808 04:53:16.640141 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.640148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.640213 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.640318 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:16.641526 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.642462 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.643576 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.644034 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.644162 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.646920 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:16.646927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:16.646932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:16.646970 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:16.648751 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.649249 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.649253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.649255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.649348 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.649664 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.649667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.649669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.649889 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:16.649987 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.650430 (+ 443us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.650561 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:53:16.651466 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.638061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:50287)
0808 04:53:16.638544 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:16.638597 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:16.639921 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:16.640535 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:16.640542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:16.640902 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:16.641368 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.641381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.642594 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.642596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.643450 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.643458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.644194 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.644197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.644349 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.645027 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:16.645044 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:16.646731 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:16.648869 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.648873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.648884 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.649150 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.649442 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.649444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.649445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.649576 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.649988 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:16.649993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:16.650205 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 04:53:16.650359 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:53:16.650600 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":143,"threads_started":1}
I20260808 04:53:16.652658 6029 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42771
I20260808 04:53:16.652913 6029 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:16.653393 6029 heartbeater.cc:511] Master 127.15.30.126:42771 requested a full tablet report, sending...
I20260808 04:53:16.654414 5525 ts_manager.cc:194] Registered new tserver with Master: 987742573cea4391a897720d3a1d263a (127.15.30.66:41817)
I20260808 04:53:16.654981 5525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:50287
W20260808 04:53:16.755347 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:16.755573 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:16.755613 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:16.759572 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:16.759632 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:16.759718 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:53:16.764904 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:37833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=41947
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:16.765949 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:16.766959 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:16.776659 6054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:16.776633 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:16.776639 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:16.777062 6036 server_base.cc:1034] running on GCE node
I20260808 04:53:16.777426 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:16.778122 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:16.779300 6036 hybrid_clock.cc:648] HybridClock initialized: now 1786164796779279 us; error 41 us; skew 500 ppm
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:16.782045 6036 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:53:16.782823 6036 webserver.cc:460] Webserver started at http://127.15.30.67:41947/ using document root <none> and password file <none>
I20260808 04:53:16.783409 6036 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:16.783485 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:16.787153 6036 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:16.789224 6063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.790535 6036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:16.790650 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "12b521e0d9f34a3dad0b1b50dacbb1be"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "bd86b7caba91979bbeea40d188debb9b"
server_key_iv: "3eff05d45476ff83024f0f4c17682d26"
server_key_version: "encryptionkey@0"
I20260808 04:53:16.791169 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:16.804945 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:16.807658 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:16.807857 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:16.808459 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:16.809274 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:16.809322 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.809360 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:16.809409 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:16.818488 6036 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:37833
I20260808 04:53:16.818513 6234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:37833 every 8 connection(s)
I20260808 04:53:16.819435 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:53:16.823908 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 6036
Aug 08 04:53:16 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:16.833901 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.824600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.15.30.126:42771)
0808 04:53:16.824834 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.824839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.824985 (+ 146us) server_negotiation.cc:407] Connection header received
0808 04:53:16.825072 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.825077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.825136 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.825235 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:16.826119 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.826919 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.828025 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.828406 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.828496 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.830198 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:16.830205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:16.830210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:16.830241 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:16.832038 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.832559 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.832565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.832568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.832660 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.832964 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.832967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.832970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.833200 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:16.833369 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.833611 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.833730 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260808 04:53:16.834438 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.821240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:45203 (local address 127.15.30.126:42771)
0808 04:53:16.821385 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.821390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.822200 (+ 810us) server_negotiation.cc:407] Connection header received
0808 04:53:16.823494 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.823496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.823543 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.823618 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:16.825166 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.826005 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.827459 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.827807 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.827914 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.830647 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:16.830651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:16.830653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:16.830694 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:16.832172 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.832781 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.832785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.832787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.832866 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:16.833272 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:16.833277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:16.833281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:16.833482 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:16.833587 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.834182 (+ 595us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.834293 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:53:16.835289 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.821630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:45203)
0808 04:53:16.822052 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:16.822092 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:16.823303 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:16.823859 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:16.823869 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:16.824339 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:16.825011 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.825030 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.826130 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.826134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.827340 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.827349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.827947 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.827950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.828151 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.828816 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:16.828832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:16.830486 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:16.832282 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.832286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.832297 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.832672 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.832960 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:16.832962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:16.832964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:16.833144 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:16.833597 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:16.833604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:16.833882 (+ 278us) client_negotiation.cc:769] Sending connection context
0808 04:53:16.834111 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 04:53:16.834427 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260808 04:53:16.836315 6236 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42771
I20260808 04:53:16.836570 6236 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:16.837026 6236 heartbeater.cc:511] Master 127.15.30.126:42771 requested a full tablet report, sending...
I20260808 04:53:16.838023 5525 ts_manager.cc:194] Registered new tserver with Master: 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
I20260808 04:53:16.838610 5525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:45203
I20260808 04:53:16.847337 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:53:16.851130 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.847915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.15.30.126:42771)
0808 04:53:16.848031 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.848034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.848221 (+ 187us) server_negotiation.cc:407] Connection header received
0808 04:53:16.848332 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.848335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.848482 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.848564 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:53:16.848770 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.849247 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.850030 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.850224 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.850293 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.850545 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:53:16.850867 (+ 322us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:53:16.850949 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.851009 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.851046 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:53:16.851753 5525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:53:16.854969 5525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:53:16.862812 5725 tablet_service.cc:1467] Processing CreateTablet for tablet 09c2430f4f73499f983331df55db4fe0 (DEFAULT_TABLE table=test-table [id=c705433083ca48a399ed673ac9a06eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:16.863477 5725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c2430f4f73499f983331df55db4fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:16.866321 5655 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap starting.
I20260808 04:53:16.867223 5655 tablet_bootstrap.cc:654] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:16.867473 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.861963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37833 (local address 127.0.0.1:54662)
0808 04:53:16.862524 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:16.862538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:16.862661 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:16.863144 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:16.863147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:16.863158 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:16.863403 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.863409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.864724 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.864728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.866020 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.866031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.867057 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.867061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.867146 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.867186 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 04:53:16.867263 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:53:16.867337 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260808 04:53:16.867507 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.862139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.15.30.67:37833)
0808 04:53:16.862471 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.862483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.862656 (+ 173us) server_negotiation.cc:407] Connection header received
0808 04:53:16.862769 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.862774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.862960 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.863114 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:16.863527 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.864574 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.866204 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.866927 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.867055 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.867205 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.867293 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.867379 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
I20260808 04:53:16.867707 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.861622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41817 (local address 127.0.0.1:34898)
0808 04:53:16.861746 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:16.861761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:16.861843 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:16.863116 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:16.863121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:16.863142 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:16.863266 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.863273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.864967 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.864970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.866103 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:16.866115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.867244 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.867249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:16.867356 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.867388 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:53:16.867446 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 04:53:16.867543 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":19,"spinlock_wait_cycles":2688}
I20260808 04:53:16.868531 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:16.862118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.15.30.66:41817)
0808 04:53:16.862575 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:16.862587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:16.862613 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:53:16.862699 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:16.862707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:16.862908 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:16.863273 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:16.863455 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.864802 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.866276 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:16.867056 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:16.867988 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:16.868161 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:16.868218 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 04:53:16.868321 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":152,"threads_started":1}
I20260808 04:53:16.869021 5655 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: No bootstrap required, opened a new log
I20260808 04:53:16.869125 5655 ts_tablet_manager.cc:1397] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 04:53:16.869565 5655 raft_consensus.cc:348] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:16.869700 5655 raft_consensus.cc:374] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:16.869742 5655 raft_consensus.cc:729] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0468efd23240ec9ed81bb8dc8ef345, State: Initialized, Role: FOLLOWER
I20260808 04:53:16.869902 5655 consensus_queue.cc:260] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:16.870255 5655 ts_tablet_manager.cc:1428] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:53:16.870692 6139 tablet_service.cc:1467] Processing CreateTablet for tablet 09c2430f4f73499f983331df55db4fe0 (DEFAULT_TABLE table=test-table [id=c705433083ca48a399ed673ac9a06eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:16.871271 5938 tablet_service.cc:1467] Processing CreateTablet for tablet 09c2430f4f73499f983331df55db4fe0 (DEFAULT_TABLE table=test-table [id=c705433083ca48a399ed673ac9a06eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:16.871660 6139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c2430f4f73499f983331df55db4fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:16.872256 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c2430f4f73499f983331df55db4fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:16.878942 6254 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Bootstrap starting.
I20260808 04:53:16.879387 6255 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Bootstrap starting.
I20260808 04:53:16.881068 6254 tablet_bootstrap.cc:654] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:16.881703 6255 tablet_bootstrap.cc:654] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:16.882438 6254 log.cc:824] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:16.882593 6255 log.cc:824] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:16.884580 6255 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: No bootstrap required, opened a new log
I20260808 04:53:16.884583 6254 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: No bootstrap required, opened a new log
I20260808 04:53:16.884784 6255 ts_tablet_manager.cc:1397] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:16.884784 6254 ts_tablet_manager.cc:1397] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:16.888769 6255 raft_consensus.cc:348] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:16.888769 6254 raft_consensus.cc:348] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:16.889050 6255 raft_consensus.cc:374] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:16.889050 6254 raft_consensus.cc:374] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:16.889117 6254 raft_consensus.cc:729] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b521e0d9f34a3dad0b1b50dacbb1be, State: Initialized, Role: FOLLOWER
I20260808 04:53:16.889117 6255 raft_consensus.cc:729] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987742573cea4391a897720d3a1d263a, State: Initialized, Role: FOLLOWER
I20260808 04:53:16.889587 6254 consensus_queue.cc:260] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:16.889590 6255 consensus_queue.cc:260] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:16.890349 6029 heartbeater.cc:503] Master 127.15.30.126:42771 was elected leader, sending a full tablet report...
I20260808 04:53:16.890583 6236 heartbeater.cc:503] Master 127.15.30.126:42771 was elected leader, sending a full tablet report...
I20260808 04:53:16.890633 6255 ts_tablet_manager.cc:1428] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:16.890618 6254 ts_tablet_manager.cc:1428] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260808 04:53:16.935397 5821 tablet.cc:2369] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:53:17.070449 6237 tablet.cc:2369] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:53:17.136864 6031 tablet.cc:2369] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:53:17.140035 5830 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:17.140276 5830 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:17.141539 5830 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
I20260808 04:53:17.145731 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.141947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:60753 (local address 127.15.30.67:37833)
0808 04:53:17.142076 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:17.142081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:17.142416 (+ 335us) server_negotiation.cc:407] Connection header received
0808 04:53:17.142544 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:17.142547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:17.142598 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:17.142682 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:17.142990 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.143472 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.144627 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.145065 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.145169 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:17.145253 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:17.145509 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 04:53:17.145592 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:53:17.145754 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.141787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41817 (local address 127.15.30.65:44011)
0808 04:53:17.142115 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.142137 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:17.142249 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:17.142557 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:17.142559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:17.142578 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:17.142718 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:17.142726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.143584 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.143589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:17.144545 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:17.144556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.145272 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.145279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:17.145360 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:17.145428 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 04:53:17.145525 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 04:53:17.145594 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":124,"thread_start_us":85,"threads_started":1}
I20260808 04:53:17.145761 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.141869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:44011 (local address 127.15.30.66:41817)
0808 04:53:17.142007 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:17.142013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:17.142229 (+ 216us) server_negotiation.cc:407] Connection header received
0808 04:53:17.142385 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:17.142388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:17.142444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:17.142527 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:17.142854 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.143454 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.144712 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.145129 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.145251 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:17.145315 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:17.145566 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 04:53:17.145647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:53:17.145761 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.141814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37833 (local address 127.15.30.65:60753)
0808 04:53:17.142273 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.142293 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:17.142414 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:17.142716 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:17.142718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:17.142735 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:17.142869 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:17.142876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.143620 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.143623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:17.144504 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:17.144513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.145212 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.145216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:17.145322 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:17.145400 (+ 78us) client_negotiation.cc:769] Sending connection context
0808 04:53:17.145462 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 04:53:17.145558 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":46,"thread_start_us":67,"threads_started":1}
I20260808 04:53:17.146449 5966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c2430f4f73499f983331df55db4fe0" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987742573cea4391a897720d3a1d263a" is_pre_election: true
I20260808 04:53:17.146553 6172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c2430f4f73499f983331df55db4fe0" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" is_pre_election: true
I20260808 04:53:17.146801 6172 raft_consensus.cc:2454] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 0.
I20260808 04:53:17.146801 5966 raft_consensus.cc:2454] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 0.
I20260808 04:53:17.147332 5650 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0468efd23240ec9ed81bb8dc8ef345, 987742573cea4391a897720d3a1d263a; no voters:
I20260808 04:53:17.147653 5830 raft_consensus.cc:2781] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:53:17.147742 5830 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:53:17.147801 5830 raft_consensus.cc:3037] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:17.148900 5830 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:17.149293 5830 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 election: Requested vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
I20260808 04:53:17.149717 5966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c2430f4f73499f983331df55db4fe0" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "987742573cea4391a897720d3a1d263a"
I20260808 04:53:17.149705 6172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c2430f4f73499f983331df55db4fe0" candidate_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be"
I20260808 04:53:17.149843 5966 raft_consensus.cc:3037] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:17.149843 6172 raft_consensus.cc:3037] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:17.151171 6172 raft_consensus.cc:2454] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 1.
I20260808 04:53:17.151166 5966 raft_consensus.cc:2454] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0468efd23240ec9ed81bb8dc8ef345 in term 1.
I20260808 04:53:17.151485 5645 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 3d0468efd23240ec9ed81bb8dc8ef345; no voters:
I20260808 04:53:17.151686 5830 raft_consensus.cc:2781] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:17.151877 5830 raft_consensus.cc:686] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0468efd23240ec9ed81bb8dc8ef345, State: Running, Role: LEADER
I20260808 04:53:17.152154 5830 consensus_queue.cc:237] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:17.155449 5524 catalog_manager.cc:5495] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0468efd23240ec9ed81bb8dc8ef345 (127.15.30.65). New cstate: current_term: 1 leader_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } health_report { overall_health: HEALTHY } } }
I20260808 04:53:17.234277 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.230104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.15.30.65:46303)
0808 04:53:17.230499 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:17.230508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:17.230526 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:53:17.230620 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:17.230623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:17.230696 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:17.230834 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:53:17.231078 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.231843 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.232702 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:17.233059 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:17.233156 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:17.233400 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:53:17.233890 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:53:17.233998 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:17.234059 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 04:53:17.234113 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":167,"threads_started":1}
I20260808 04:53:17.238521 5966 raft_consensus.cc:1261] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Refusing update from remote peer 3d0468efd23240ec9ed81bb8dc8ef345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:17.238521 6172 raft_consensus.cc:1261] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Refusing update from remote peer 3d0468efd23240ec9ed81bb8dc8ef345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:17.239332 5830 consensus_queue.cc:1035] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:17.239543 6264 consensus_queue.cc:1035] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:17.246230 6268 mvcc.cc:205] Tried to move back new op lower bound from 7316131009482260480 to 7316131009137954816. Current Snapshot: MvccSnapshot[applied={T|T < 7316131009482260480}]
I20260808 04:53:17.247259 6270 mvcc.cc:205] Tried to move back new op lower bound from 7316131009482260480 to 7316131009137954816. Current Snapshot: MvccSnapshot[applied={T|T < 7316131009482260480}]
I20260808 04:53:17.255717 5525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:17.255892 5525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:17.258490 5525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Sending DeleteTablet for 3 replicas of tablet 09c2430f4f73499f983331df55db4fe0
I20260808 04:53:17.259163 5938 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c2430f4f73499f983331df55db4fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:17 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:34898
I20260808 04:53:17.259502 5725 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c2430f4f73499f983331df55db4fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:17 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:47602
I20260808 04:53:17.259869 6287 tablet_replica.cc:330] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: stopping tablet replica
I20260808 04:53:17.259991 6287 raft_consensus.cc:2229] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:53:17.260239 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 5584
I20260808 04:53:17.275940 6139 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c2430f4f73499f983331df55db4fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:17 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:54662
I20260808 04:53:17.276114 6286 tablet_replica.cc:330] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: stopping tablet replica
I20260808 04:53:17.276501 6289 tablet_replica.cc:330] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: stopping tablet replica
I20260808 04:53:17.276566 6286 raft_consensus.cc:2229] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:17.276816 6289 raft_consensus.cc:2229] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:17.276957 6286 raft_consensus.cc:2258] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:17.277165 6289 raft_consensus.cc:2258] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:17.278524 6289 ts_tablet_manager.cc:1905] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:17.278650 6286 ts_tablet_manager.cc:1905] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:17.280869 6289 ts_tablet_manager.cc:1918] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:17.280999 6289 log.cc:1196] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/09c2430f4f73499f983331df55db4fe0
I20260808 04:53:17.281378 6289 ts_tablet_manager.cc:1939] T 09c2430f4f73499f983331df55db4fe0 P 12b521e0d9f34a3dad0b1b50dacbb1be: Deleting consensus metadata
I20260808 04:53:17.282119 6286 ts_tablet_manager.cc:1918] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:17.282249 6286 log.cc:1196] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/09c2430f4f73499f983331df55db4fe0
I20260808 04:53:17.282590 6286 ts_tablet_manager.cc:1939] T 09c2430f4f73499f983331df55db4fe0 P 987742573cea4391a897720d3a1d263a: Deleting consensus metadata
I20260808 04:53:17.282545 5499 catalog_manager.cc:4841] TS 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): tablet 09c2430f4f73499f983331df55db4fe0 (table test-table [id=c705433083ca48a399ed673ac9a06eb6]) successfully deleted
W20260808 04:53:17.283336 5505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:46303 (error 108)
I20260808 04:53:17.283847 5505 catalog_manager.cc:4841] TS 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): tablet 09c2430f4f73499f983331df55db4fe0 (table test-table [id=c705433083ca48a399ed673ac9a06eb6]) successfully deleted
I20260808 04:53:17.284142 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 5828
I20260808 04:53:17.288405 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.284441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:46303 (local address 127.0.0.1:47616)
0808 04:53:17.288210 (+ 3769us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.288289 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3657}
W20260808 04:53:17.289994 5505 catalog_manager.cc:4584] TS 3d0468efd23240ec9ed81bb8dc8ef345 (127.15.30.65:46303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09c2430f4f73499f983331df55db4fe0: Network error: Client connection negotiation failed: client connection to 127.15.30.65:46303: connect: Connection refused (error 111)
I20260808 04:53:17.296212 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 6036
I20260808 04:53:17.308930 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 5465
I20260808 04:53:17.321338 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42771
--webserver_interface=127.15.30.126
--webserver_port=37283
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:42771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:17.431087 6292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:17.431310 6292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:17.431348 6292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:17.435225 6292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:17.435279 6292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:17.435298 6292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:17.435314 6292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:17.435329 6292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:17.440297 6292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:42771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:17.441208 6292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:17.442087 6292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:17.450569 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:17.450677 6292 server_base.cc:1034] running on GCE node
W20260808 04:53:17.450599 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:17.450599 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:17.451390 6292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:17.452450 6292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:17.453632 6292 hybrid_clock.cc:648] HybridClock initialized: now 1786164797453608 us; error 47 us; skew 500 ppm
Aug 08 04:53:17 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:17.456245 6292 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:17.457106 6292 webserver.cc:460] Webserver started at http://127.15.30.126:37283/ using document root <none> and password file <none>
I20260808 04:53:17.457597 6292 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:17.457656 6292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:17.461261 6292 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260808 04:53:17.463239 6311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:17.464489 6292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:53:17.464610 6292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "1b06e201285443e78c2690872b8663ed"
format_stamp: "Formatted at 2026-08-08 04:53:14 on dist-test-slave-zpvx"
server_key: "f5310e19b62e307db4803b060acdb733"
server_key_iv: "b9b10d1c8da7c0d43b904419d227254a"
server_key_version: "encryptionkey@0"
I20260808 04:53:17.465123 6292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:17.482676 6292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:17.486020 6292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:17.486243 6292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:17.500967 6292 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:42771
I20260808 04:53:17.501336 6392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:42771 every 8 connection(s)
I20260808 04:53:17.502241 6292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:17.506534 6396 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:53:17.507798 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 6292
I20260808 04:53:17.508277 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:46303
--local_ip_for_outbound_sockets=127.15.30.65
--tserver_master_addrs=127.15.30.126:42771
--webserver_port=46117
--webserver_interface=127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:17.508528 6396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Bootstrap starting.
I20260808 04:53:17.527271 6396 log.cc:824] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:17.538563 6396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:53:17.538965 6396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed: Bootstrap complete.
I20260808 04:53:17.543080 6396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:17.543493 6396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b06e201285443e78c2690872b8663ed, State: Initialized, Role: FOLLOWER
I20260808 04:53:17.544005 6396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:17.544121 6396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:17.544212 6396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:17.544338 6396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:53:17.546233 6396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:17.546566 6396 leader_election.cc:302] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b06e201285443e78c2690872b8663ed; no voters:
I20260808 04:53:17.546923 6396 leader_election.cc:288] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:53:17.547005 6400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:53:17.547197 6400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [term 3 LEADER]: Becoming Leader. State: Replica: 1b06e201285443e78c2690872b8663ed, State: Running, Role: LEADER
I20260808 04:53:17.547483 6400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } }
I20260808 04:53:17.548143 6396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:17.548441 6400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b06e201285443e78c2690872b8663ed. Latest consensus state: current_term: 3 leader_uuid: "1b06e201285443e78c2690872b8663ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } } }
I20260808 04:53:17.548565 6400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:17.548962 6401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1b06e201285443e78c2690872b8663ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b06e201285443e78c2690872b8663ed" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42771 } } }
I20260808 04:53:17.549149 6401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b06e201285443e78c2690872b8663ed [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:17.550220 6407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:17.555615 6407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:17.556222 6407 catalog_manager.cc:1259] Loaded cluster ID: 2e3b32adcd3d44228b107b108f27f0a0
I20260808 04:53:17.556280 6407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 04:53:17.557359 6407 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6a3a4b59ad google::LogMessage::Fail() at ??:0
@ 0x7f6a3a4b888a google::LogMessage::SendToLog() at ??:0
@ 0x7f6a3a4b54de google::LogMessage::Flush() at ??:0
@ 0x7f6a3a4b6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6a44df4fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6a44df5d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6a44df2579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6a44e36640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6a3c7073d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6a3c707c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6a3c7093f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6a3c6f8512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6a3dd3c6db start_thread at ??:0
@ 0x7f6a38fb361f clone at ??:0
W20260808 04:53:17.654677 6398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:17.655026 6398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:17.655089 6398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:17.661271 6398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:17.661350 6398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:17.661434 6398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:53:17.669137 6398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:46303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=46117
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:17.670444 6398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:17.671772 6398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:17.682940 6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:17.683081 6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:17.682998 6459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:17.683310 6398 server_base.cc:1034] running on GCE node
I20260808 04:53:17.683838 6398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:17.684656 6398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:17.685851 6398 hybrid_clock.cc:648] HybridClock initialized: now 1786164797685818 us; error 52 us; skew 500 ppm
Aug 08 04:53:17 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:17.688658 6398 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:53:17.689564 6398 webserver.cc:460] Webserver started at http://127.15.30.65:46117/ using document root <none> and password file <none>
I20260808 04:53:17.690122 6398 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:17.690186 6398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:17.694052 6398 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:53:17.696394 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:17.697703 6398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:53:17.697841 6398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "3d0468efd23240ec9ed81bb8dc8ef345"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "398d4dc7407ff7169ab8c911651965f7"
server_key_iv: "dcb646002d2c6d16d8f990c3b5bdc77e"
server_key_version: "encryptionkey@0"
I20260808 04:53:17.698400 6398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:17.726089 6398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:17.729549 6398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:17.729764 6398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:17.730433 6398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:17.731577 6487 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:53:17.735975 6398 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:53:17.736037 6398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260808 04:53:17.736083 6398 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:53:17.737807 6398 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:53:17.737865 6398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:53:17.738133 6487 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap starting.
I20260808 04:53:17.748777 6398 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:46303
I20260808 04:53:17.749043 6637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:46303 every 8 connection(s)
I20260808 04:53:17.750005 6398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:53:17.756961 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 6398
I20260808 04:53:17.757441 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:41817
--local_ip_for_outbound_sockets=127.15.30.66
--tserver_master_addrs=127.15.30.126:42771
--webserver_port=40303
--webserver_interface=127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:17.758852 6487 log.cc:824] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:17.768014 6487 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:53:17.768482 6487 tablet_bootstrap.cc:492] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Bootstrap complete.
I20260808 04:53:17.770001 6487 ts_tablet_manager.cc:1397] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.006s
I20260808 04:53:17.773304 6487 raft_consensus.cc:348] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:17.773865 6487 raft_consensus.cc:729] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0468efd23240ec9ed81bb8dc8ef345, State: Initialized, Role: FOLLOWER
I20260808 04:53:17.774386 6487 consensus_queue.cc:260] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:17.775447 6487 ts_tablet_manager.cc:1428] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
W20260808 04:53:17.793960 6641 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:53:17.752430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:53853)
0808 04:53:17.753014 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.753073 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:17.754164 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:17.793160 (+ 38996us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":154,"threads_started":1}
W20260808 04:53:17.794512 6639 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:42771 (0 consecutive failures): Network error: Failed to ping master at 127.15.30.126:42771: Client connection negotiation failed: client connection to 127.15.30.126:42771: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:53:17.795289 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.795039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:57309)
0808 04:53:17.795168 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.795234 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 04:53:17.796051 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.795795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:33643)
0808 04:53:17.795936 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.795989 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 04:53:17.796433 6639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:53:17.880393 6646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:17.880637 6646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:17.880679 6646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:17.884646 6646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:17.884696 6646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:17.884792 6646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:53:17.889834 6646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:41817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=40303
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:17.890810 6646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:17.891815 6646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:17.901141 6657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:17.901324 6656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:17.901374 6646 server_base.cc:1034] running on GCE node
W20260808 04:53:17.901368 6660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:17.901909 6646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:17.902606 6646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:17.903795 6646 hybrid_clock.cc:648] HybridClock initialized: now 1786164797903780 us; error 32 us; skew 500 ppm
Aug 08 04:53:17 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:17.906517 6646 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:53:17.907374 6646 webserver.cc:460] Webserver started at http://127.15.30.66:40303/ using document root <none> and password file <none>
I20260808 04:53:17.907919 6646 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:17.908012 6646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:17.911731 6646 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:17.913836 6671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:17.915043 6646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 04:53:17.915184 6646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "987742573cea4391a897720d3a1d263a"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "80f6c5e8b3da48843b0f831d4301c991"
server_key_iv: "9d9db61dfefad29c6e25a2f56ef27b16"
server_key_version: "encryptionkey@0"
I20260808 04:53:17.915710 6646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:17.931653 6646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:17.934299 6646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:17.934489 6646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:17.935038 6646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:17.935827 6646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:17.935902 6646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:17.935961 6646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:17.936014 6646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:17.944964 6646 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:41817
I20260808 04:53:17.945001 6839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:41817 every 8 connection(s)
I20260808 04:53:17.945930 6646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:53:17.946429 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 6646
I20260808 04:53:17.946936 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:37833
--local_ip_for_outbound_sockets=127.15.30.67
--tserver_master_addrs=127.15.30.126:42771
--webserver_port=41947
--webserver_interface=127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:17.950117 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.948019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:59929)
0808 04:53:17.948598 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.948786 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":117,"threads_started":1}
W20260808 04:53:17.950591 6840 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:42771 (0 consecutive failures): Network error: Failed to ping master at 127.15.30.126:42771: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
I20260808 04:53:17.951295 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.950993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:42047)
0808 04:53:17.951142 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.951210 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 04:53:17.952033 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:17.951759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:48229)
0808 04:53:17.951901 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:17.951962 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:53:17.952239 6840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:53:18.061077 6849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:18.061340 6849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:18.061383 6849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:18.065246 6849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:18.065294 6849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:18.065346 6849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:53:18.070623 6849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:37833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=41947
--tserver_master_addrs=127.15.30.126:42771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:18.071571 6849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:18.072573 6849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:18.082319 6861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:18.082476 6859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:18.082352 6858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:18.082559 6849 server_base.cc:1034] running on GCE node
I20260808 04:53:18.082993 6849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:18.083606 6849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:18.084798 6849 hybrid_clock.cc:648] HybridClock initialized: now 1786164798084770 us; error 41 us; skew 500 ppm
Aug 08 04:53:18 dist-test-slave-zpvx krb5kdc[4593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164798, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:18.087462 6849 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:53:18.088287 6849 webserver.cc:460] Webserver started at http://127.15.30.67:41947/ using document root <none> and password file <none>
I20260808 04:53:18.088866 6849 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:18.088958 6849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:18.092756 6849 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 04:53:18.094878 6871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:18.096107 6849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:18.096267 6849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "12b521e0d9f34a3dad0b1b50dacbb1be"
format_stamp: "Formatted at 2026-08-08 04:53:15 on dist-test-slave-zpvx"
server_key: "bd86b7caba91979bbeea40d188debb9b"
server_key_iv: "3eff05d45476ff83024f0f4c17682d26"
server_key_version: "encryptionkey@0"
I20260808 04:53:18.096900 6849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:18.111265 6849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:18.114254 6849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:18.114464 6849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:18.115015 6849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:18.115782 6849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:18.115830 6849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:18.115869 6849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:18.115892 6849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:18.125190 6849 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:37833
I20260808 04:53:18.125223 7055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:37833 every 8 connection(s)
I20260808 04:53:18.126175 6849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:53:18.130119 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:18.128275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:45507)
0808 04:53:18.128859 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:18.128975 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":133,"threads_started":1}
W20260808 04:53:18.130610 7056 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:42771 (0 consecutive failures): Network error: Failed to ping master at 127.15.30.126:42771: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
I20260808 04:53:18.131150 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:18.130944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:40985)
0808 04:53:18.131052 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:18.131096 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 04:53:18.131785 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:18.131565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:54749)
0808 04:53:18.131673 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:18.131729 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:53:18.131965 7056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:53:18.134788 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 6849
I20260808 04:53:18.797433 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:18.797004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:50887)
0808 04:53:18.797265 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:18.797320 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260808 04:53:18.953111 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:18.952743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:52477)
0808 04:53:18.952965 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:18.953014 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20260808 04:53:19.017237 7065 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:19.017498 7065 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:19.018628 7065 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
Aug 08 04:53:19 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.66@KRBTEST.COM
Aug 08 04:53:19 dist-test-slave-zpvx krb5kdc[4593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.67@KRBTEST.COM
I20260808 04:53:19.038786 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:19.018955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:37833 (local address 127.15.30.65:60437)
0808 04:53:19.019087 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:19.019107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:19.019198 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:19.021112 (+ 1914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:19.021118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:19.021666 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:19.022132 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:19.022145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.023408 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.023416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:19.024368 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:19.024399 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.025376 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.025380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:19.025671 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:19.031140 (+ 5469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:19.031167 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:19.033514 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:19.035947 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:19.035952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:19.035958 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:19.036243 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:19.036631 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:19.036637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:19.036641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:19.036781 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:19.037574 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:19.037581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:19.037949 (+ 368us) client_negotiation.cc:769] Sending connection context
0808 04:53:19.038180 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 04:53:19.038625 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":367,"spinlock_wait_cycles":21888}
I20260808 04:53:19.038827 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:19.019126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:58533 (local address 127.15.30.66:41817)
0808 04:53:19.019553 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:19.019579 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:19.019729 (+ 150us) server_negotiation.cc:407] Connection header received
0808 04:53:19.020607 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:19.020617 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:19.020909 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:19.021186 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:19.022315 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.023255 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.024568 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.025246 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.025423 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:19.033361 (+ 7938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:19.033370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:19.033388 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:19.033418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:19.035585 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:19.036249 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:19.036253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:19.036257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:19.036465 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:19.036916 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:19.036920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:19.036922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:19.037442 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:19.037589 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:19.038328 (+ 739us) server_negotiation.cc:299] Negotiation successful
0808 04:53:19.038635 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260808 04:53:19.038825 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:19.019126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:60437 (local address 127.15.30.67:37833)
0808 04:53:19.019463 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:19.019539 (+ 76us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:19.019588 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:53:19.020331 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:19.020338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:19.020758 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:19.021061 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:19.022308 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.023264 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.024518 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.025241 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.025422 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:19.033718 (+ 8296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:19.033727 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:19.033745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:19.033782 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:19.035843 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:19.036347 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:19.036352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:19.036357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:19.036516 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:19.036916 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:19.036920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:19.036922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:19.037442 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:19.037589 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:19.038328 (+ 739us) server_negotiation.cc:299] Negotiation successful
0808 04:53:19.038635 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260808 04:53:19.038786 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:19.018955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41817 (local address 127.15.30.65:58533)
0808 04:53:19.019591 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:19.019608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:19.019749 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:19.021130 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:19.021133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:19.021671 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:19.022132 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:19.022146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.023411 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.023415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:19.024344 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:19.024354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:19.025376 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:19.025380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:19.025668 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:19.031139 (+ 5471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:19.031167 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:19.033119 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:19.035722 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:19.035727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:19.035738 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:19.036125 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:19.036569 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:19.036572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:19.036573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:19.036781 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:19.037574 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:19.037581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:19.037949 (+ 368us) client_negotiation.cc:769] Sending connection context
0808 04:53:19.038188 (+ 239us) client_negotiation.cc:240] Negotiation successful
0808 04:53:19.038567 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":596,"spinlock_wait_cycles":74752,"thread_start_us":86,"threads_started":1}
W20260808 04:53:19.041651 6483 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
W20260808 04:53:19.041916 6479 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
I20260808 04:53:19.042022 6479 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0468efd23240ec9ed81bb8dc8ef345; no voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 987742573cea4391a897720d3a1d263a
I20260808 04:53:19.042271 7065 raft_consensus.cc:2726] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:53:19.132819 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:19.132457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:45649)
0808 04:53:19.132673 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:19.132732 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260808 04:53:19.798578 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:19.798174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:57965)
0808 04:53:19.798423 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:19.798479 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260808 04:53:19.954149 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:19.953784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:47509)
0808 04:53:19.954011 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:19.954063 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260808 04:53:20.133913 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:20.133496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:48035)
0808 04:53:20.133754 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:20.133808 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260808 04:53:20.799782 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:20.799348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:40719)
0808 04:53:20.799625 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:20.799682 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
I20260808 04:53:20.955207 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:20.954833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:54703)
0808 04:53:20.955067 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:20.955121 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260808 04:53:21.135048 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:21.134607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:45885)
0808 04:53:21.134882 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:21.134943 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260808 04:53:21.221140 7079 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:21.221290 7079 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:21.221712 7079 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
W20260808 04:53:21.222512 6479 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
W20260808 04:53:21.222608 6483 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
I20260808 04:53:21.222723 6483 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0468efd23240ec9ed81bb8dc8ef345; no voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 987742573cea4391a897720d3a1d263a
I20260808 04:53:21.222884 7079 raft_consensus.cc:2726] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:53:21.801137 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:21.800647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:51755)
0808 04:53:21.800980 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:21.801037 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":138,"threads_started":1}
I20260808 04:53:21.956594 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:21.956007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:34167)
0808 04:53:21.956334 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:21.956448 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260808 04:53:22.136231 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:22.135811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:52613)
0808 04:53:22.136058 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:22.136140 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260808 04:53:22.802747 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:22.802249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:49593)
0808 04:53:22.802594 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:22.802648 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260808 04:53:22.957767 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:22.957341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:53425)
0808 04:53:22.957604 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:22.957657 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260808 04:53:23.137403 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:23.136995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:57541)
0808 04:53:23.137258 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:23.137310 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260808 04:53:23.803941 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:23.803544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:58753)
0808 04:53:23.803784 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:23.803839 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260808 04:53:23.949318 7090 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:23.949478 7090 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:23.949906 7090 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
W20260808 04:53:23.950747 6483 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
W20260808 04:53:23.950856 6479 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
I20260808 04:53:23.950923 6479 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0468efd23240ec9ed81bb8dc8ef345; no voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 987742573cea4391a897720d3a1d263a
I20260808 04:53:23.951061 7090 raft_consensus.cc:2726] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:53:23.958933 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:23.958553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:53411)
0808 04:53:23.958795 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:23.958847 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260808 04:53:24.138653 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:24.138207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:42375)
0808 04:53:24.138491 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:24.138561 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260808 04:53:24.805120 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:24.804713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:44327)
0808 04:53:24.804964 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:24.805020 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260808 04:53:24.960073 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:24.959641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:54477)
0808 04:53:24.959927 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:24.959982 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":72,"threads_started":1}
I20260808 04:53:25.139875 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:25.139442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:54851)
0808 04:53:25.139729 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:25.139781 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":70,"threads_started":1}
I20260808 04:53:25.806313 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:25.805896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:38167)
0808 04:53:25.806156 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:25.806213 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260808 04:53:25.961282 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:25.960882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:39751)
0808 04:53:25.961128 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:25.961185 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260808 04:53:26.141005 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:26.140591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:53545)
0808 04:53:26.140858 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:26.140913 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260808 04:53:26.626972 7105 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:26.627173 7105 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:26.627581 7105 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
W20260808 04:53:26.628456 6479 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
W20260808 04:53:26.628629 6483 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
I20260808 04:53:26.628734 6483 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0468efd23240ec9ed81bb8dc8ef345; no voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 987742573cea4391a897720d3a1d263a
I20260808 04:53:26.628925 7105 raft_consensus.cc:2726] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:53:26.807487 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:26.807079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:40483)
0808 04:53:26.807334 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:26.807388 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260808 04:53:26.962495 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:26.962054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:57961)
0808 04:53:26.962351 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:26.962401 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":121,"threads_started":1}
I20260808 04:53:27.142100 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:27.141704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:40739)
0808 04:53:27.141951 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:27.142004 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260808 04:53:27.808779 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:27.808337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:56705)
0808 04:53:27.808615 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:27.808670 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260808 04:53:27.963712 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:27.963280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:49951)
0808 04:53:27.963564 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:27.963619 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
I20260808 04:53:28.143333 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:28.142884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:58799)
0808 04:53:28.143129 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:28.143211 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260808 04:53:28.809937 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:28.809514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:51061)
0808 04:53:28.809786 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:28.809839 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
I20260808 04:53:28.964871 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:28.964424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:42687)
0808 04:53:28.964699 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:28.964757 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260808 04:53:29.144616 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:29.144132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:54919)
0808 04:53:29.144460 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:29.144516 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260808 04:53:29.711344 7121 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:29.711476 7121 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:29.711975 7121 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
W20260808 04:53:29.712795 6483 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
W20260808 04:53:29.712934 6479 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
I20260808 04:53:29.713024 6479 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0468efd23240ec9ed81bb8dc8ef345; no voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 987742573cea4391a897720d3a1d263a
I20260808 04:53:29.713212 7121 raft_consensus.cc:2726] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:53:29.811333 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:29.810868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:49559)
0808 04:53:29.811154 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:29.811234 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
I20260808 04:53:29.966190 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:29.965673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:42893)
0808 04:53:29.965956 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:29.966019 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260808 04:53:30.145794 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:30.145367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:51371)
0808 04:53:30.145645 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:30.145702 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260808 04:53:30.812582 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:30.812094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:55139)
0808 04:53:30.812406 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:30.812474 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260808 04:53:30.967414 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:30.967000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:45177)
0808 04:53:30.967265 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:30.967321 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260808 04:53:31.146998 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:31.146551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:34257)
0808 04:53:31.146836 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:31.146891 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260808 04:53:31.578974 7130 raft_consensus.cc:482] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:31.579115 7130 raft_consensus.cc:504] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987742573cea4391a897720d3a1d263a" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41817 } } peers { permanent_uuid: "12b521e0d9f34a3dad0b1b50dacbb1be" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 37833 } } peers { permanent_uuid: "3d0468efd23240ec9ed81bb8dc8ef345" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 46303 } }
I20260808 04:53:31.579514 7130 leader_election.cc:288] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 987742573cea4391a897720d3a1d263a (127.15.30.66:41817), 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833)
W20260808 04:53:31.580323 6479 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12b521e0d9f34a3dad0b1b50dacbb1be (127.15.30.67:37833): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
W20260808 04:53:31.580494 6483 leader_election.cc:341] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 987742573cea4391a897720d3a1d263a (127.15.30.66:41817): Not found: Tablet not found: 09c2430f4f73499f983331df55db4fe0
I20260808 04:53:31.580588 6483 leader_election.cc:302] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0468efd23240ec9ed81bb8dc8ef345; no voters: 12b521e0d9f34a3dad0b1b50dacbb1be, 987742573cea4391a897720d3a1d263a
I20260808 04:53:31.580750 7130 raft_consensus.cc:2726] T 09c2430f4f73499f983331df55db4fe0 P 3d0468efd23240ec9ed81bb8dc8ef345 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:53:31.813859 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:31.813372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:48527)
0808 04:53:31.813677 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:31.813754 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260808 04:53:31.968742 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:31.968232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:56949)
0808 04:53:31.968593 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:31.968648 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260808 04:53:32.148208 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:32.147776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:56367)
0808 04:53:32.148051 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:32.148109 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260808 04:53:32.815227 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:32.814783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.65:53143)
0808 04:53:32.815041 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:32.815129 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260808 04:53:32.969987 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:32.969590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.66:46397)
0808 04:53:32.969839 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:32.969890 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260808 04:53:33.138828 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 6398
I20260808 04:53:33.155889 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:33.152459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42771 (local address 127.15.30.67:53745)
0808 04:53:33.155658 (+ 3199us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:33.155739 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:42771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3082,"thread_start_us":227,"threads_started":1}
I20260808 04:53:33.161551 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 6646
I20260808 04:53:33.182381 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 6849
2026-08-08T04:53:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19741 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:53:33 dist-test-slave-zpvx krb5kdc[7143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:53:33 dist-test-slave-zpvx krb5kdc[7143](info): set up 2 sockets
Aug 08 04:53:33 dist-test-slave-zpvx krb5kdc[7143](info): commencing operation
krb5kdc: starting...
W20260808 04:53:34.429281 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:53:34 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:53:34.557862 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44831
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:34.669409 7159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:34.669708 7159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:34.669778 7159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:34.673820 7159 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:53:34.673890 7159 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:53:34.673916 7159 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:53:34.673946 7159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:34.673983 7159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:34.674037 7159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:34.674094 7159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:34.674134 7159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:34.680344 7159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:34.681617 7159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:34.682643 7159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:34.692850 7167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:34.693111 7166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:34.692865 7169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:34.693449 7159 server_base.cc:1034] running on GCE node
I20260808 04:53:34.694072 7159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:34.695169 7159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:34.696414 7159 hybrid_clock.cc:648] HybridClock initialized: now 1786164814696307 us; error 86 us; skew 500 ppm
Aug 08 04:53:34 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164814, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:34.699280 7159 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:34.700145 7159 webserver.cc:460] Webserver started at http://127.15.30.126:44835/ using document root <none> and password file <none>
I20260808 04:53:34.700780 7159 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:34.700876 7159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:34.701136 7159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:34.703202 7159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "846de0155ad1466184e63b32f1268d18"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "35c0436360f4d849d82cee04218c000d"
server_key_iv: "7624b4e5efe178cf3ea05511461c70d3"
server_key_version: "encryptionkey@0"
I20260808 04:53:34.703787 7159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "846de0155ad1466184e63b32f1268d18"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "35c0436360f4d849d82cee04218c000d"
server_key_iv: "7624b4e5efe178cf3ea05511461c70d3"
server_key_version: "encryptionkey@0"
I20260808 04:53:34.707628 7159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:53:34.710078 7179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:34.711308 7159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:34.711452 7159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "846de0155ad1466184e63b32f1268d18"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "35c0436360f4d849d82cee04218c000d"
server_key_iv: "7624b4e5efe178cf3ea05511461c70d3"
server_key_version: "encryptionkey@0"
I20260808 04:53:34.711561 7159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:34.741215 7159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:34.744438 7159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:34.744645 7159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:34.752384 7159 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:44831
I20260808 04:53:34.752703 7263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:44831 every 8 connection(s)
I20260808 04:53:34.753417 7159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:34.754424 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 7159
I20260808 04:53:34.754565 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:53:34.754894 15481 external_mini_cluster.cc:1428] Setting key 1fea69494adef263f206c42e0ba62a27
I20260808 04:53:34.756901 7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:34.763411 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Bootstrap starting.
Aug 08 04:53:34 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:34.765882 7264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:34.766614 7264 log.cc:824] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:34.768980 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: No bootstrap required, opened a new log
I20260808 04:53:34.770268 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:34.756328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.15.30.126:44831)
0808 04:53:34.756840 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:34.756856 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:34.756893 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:53:34.757596 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:34.757613 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:34.757929 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:34.758214 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:34.759175 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:34.760292 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:34.761585 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:34.762101 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:34.762239 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:34.765162 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:34.765167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:34.765182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:34.765210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:34.767144 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:34.767716 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:34.767722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:34.767726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:34.767856 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:34.768172 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:34.768174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:34.768175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:34.768670 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:34.768819 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:34.769216 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 04:53:34.769557 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":164,"threads_started":1}
I20260808 04:53:34.772987 7264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:34.773226 7264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:34.773289 7264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846de0155ad1466184e63b32f1268d18, State: Initialized, Role: FOLLOWER
I20260808 04:53:34.773759 7264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:34.773888 7264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:34.773972 7264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:34.774104 7264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:34.775024 7264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:34.775364 7264 leader_election.cc:302] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846de0155ad1466184e63b32f1268d18; no voters:
I20260808 04:53:34.775679 7264 leader_election.cc:288] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:53:34.775844 7271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:34.776171 7271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 LEADER]: Becoming Leader. State: Replica: 846de0155ad1466184e63b32f1268d18, State: Running, Role: LEADER
I20260808 04:53:34.776489 7271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:34.776976 7264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:34.778292 7274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846de0155ad1466184e63b32f1268d18. Latest consensus state: current_term: 1 leader_uuid: "846de0155ad1466184e63b32f1268d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } } }
I20260808 04:53:34.778342 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "846de0155ad1466184e63b32f1268d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } } }
I20260808 04:53:34.778426 7274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:34.778453 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:34.778824 7284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:34.782181 7284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:34.787947 7284 catalog_manager.cc:1347] Generated new cluster ID: c9f6e7f77f584daea01459f088ff4985
I20260808 04:53:34.788031 7284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:34.800835 7284 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:53:34.801980 7284 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:34.813445 7284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Generated new TSK 0
I20260808 04:53:34.814232 7284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:53:34.848066 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44831
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:34.960882 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:34.961149 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:34.961192 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:34.965088 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:34.965138 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:34.965197 7299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:53:34.970255 7299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:34.971203 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:34.972149 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:34.981796 7308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:34.981817 7307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:34.981797 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:34.981913 7299 server_base.cc:1034] running on GCE node
I20260808 04:53:34.982432 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:34.983048 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:34.984211 7299 hybrid_clock.cc:648] HybridClock initialized: now 1786164814984178 us; error 47 us; skew 500 ppm
Aug 08 04:53:34 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164814, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:34.987229 7299 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:53:34.988071 7299 webserver.cc:460] Webserver started at http://127.15.30.65:37769/ using document root <none> and password file <none>
I20260808 04:53:34.988708 7299 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:34.988803 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:34.989053 7299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:34.991074 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "08399aa6dda34b4ba17a77cbbb51bffc"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "a223d8a89fe3c2cd16260ab0340eadd8"
server_key_iv: "c2eb83af2494bfdd6283a3896fe36a97"
server_key_version: "encryptionkey@0"
I20260808 04:53:34.991657 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "08399aa6dda34b4ba17a77cbbb51bffc"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "a223d8a89fe3c2cd16260ab0340eadd8"
server_key_iv: "c2eb83af2494bfdd6283a3896fe36a97"
server_key_version: "encryptionkey@0"
I20260808 04:53:34.995501 7299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 04:53:34.997864 7323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:34.998987 7299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:53:34.999137 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "08399aa6dda34b4ba17a77cbbb51bffc"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "a223d8a89fe3c2cd16260ab0340eadd8"
server_key_iv: "c2eb83af2494bfdd6283a3896fe36a97"
server_key_version: "encryptionkey@0"
I20260808 04:53:34.999248 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:35.026114 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:35.029482 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:35.029697 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:35.030314 7299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:35.031164 7299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:35.031214 7299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.031286 7299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:35.031319 7299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.040007 7299 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:43533
I20260808 04:53:35.040108 7506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:43533 every 8 connection(s)
I20260808 04:53:35.040958 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:53:35.044598 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 7299
I20260808 04:53:35.044740 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:53:35.045037 15481 external_mini_cluster.cc:1428] Setting key 8809f282b5c9e8e73c0c209a1e2487f2
Aug 08 04:53:35 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164814, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:53:35.055667 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.042821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35875 (local address 127.15.30.126:44831)
0808 04:53:35.042965 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.042968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.043829 (+ 861us) server_negotiation.cc:407] Connection header received
0808 04:53:35.045418 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.045422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.045483 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.045593 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:35.047090 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.047895 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.049165 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.049411 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.049531 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.052057 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:35.052067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:35.052069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:35.052094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:35.053612 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.054137 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.054141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.054143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.054232 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.054574 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.054579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.054580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.054763 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:35.054874 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.055359 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.055490 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:53:35.056541 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.043183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:35875)
0808 04:53:35.043668 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.043711 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.045209 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.045820 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.045829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.046309 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:35.046915 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.046930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.048046 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.048049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.049024 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.049046 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.049571 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.049576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.049784 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.050396 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:35.050412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:35.051888 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:35.053740 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:35.053746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:35.053762 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:35.054026 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:35.054329 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:35.054332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:35.054334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:35.054475 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:35.054872 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:35.054879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:35.055127 (+ 248us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.055316 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.055632 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":120,"threads_started":1}
I20260808 04:53:35.057817 7507 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44831
I20260808 04:53:35.058107 7507 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:53:35.058741 7507 heartbeater.cc:511] Master 127.15.30.126:44831 requested a full tablet report, sending...
I20260808 04:53:35.060567 7207 ts_manager.cc:194] Registered new tserver with Master: 08399aa6dda34b4ba17a77cbbb51bffc (127.15.30.65:43533)
I20260808 04:53:35.061717 7207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:35875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:53:35.075508 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44831
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:35.188913 7513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:35.189182 7513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:35.189265 7513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:35.193130 7513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:35.193212 7513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:35.193305 7513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:53:35.198302 7513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:35.199254 7513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:35.200212 7513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:35.209476 7525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:35.209496 7529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:35.209805 7513 server_base.cc:1034] running on GCE node
W20260808 04:53:35.209488 7524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:35.210269 7513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:35.210809 7513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:35.211957 7513 hybrid_clock.cc:648] HybridClock initialized: now 1786164815211945 us; error 24 us; skew 500 ppm
Aug 08 04:53:35 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164815, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:35.214775 7513 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:53:35.215651 7513 webserver.cc:460] Webserver started at http://127.15.30.66:34177/ using document root <none> and password file <none>
I20260808 04:53:35.216226 7513 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:35.216291 7513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:35.216521 7513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:35.218389 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "bfb869d8a50c4226ae0ccead02b414d5"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "c7b0c2f64bc66e362e837e9f58869dda"
server_key_iv: "ba6e956b28408ff4c4b5ccdfa6ec95e5"
server_key_version: "encryptionkey@0"
I20260808 04:53:35.218892 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "bfb869d8a50c4226ae0ccead02b414d5"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "c7b0c2f64bc66e362e837e9f58869dda"
server_key_iv: "ba6e956b28408ff4c4b5ccdfa6ec95e5"
server_key_version: "encryptionkey@0"
I20260808 04:53:35.222494 7513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:53:35.224805 7538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.225987 7513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:53:35.226081 7513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "bfb869d8a50c4226ae0ccead02b414d5"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "c7b0c2f64bc66e362e837e9f58869dda"
server_key_iv: "ba6e956b28408ff4c4b5ccdfa6ec95e5"
server_key_version: "encryptionkey@0"
I20260808 04:53:35.226161 7513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:35.254688 7513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:35.257947 7513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:35.258103 7513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:35.258644 7513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:35.259425 7513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:35.259472 7513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.259513 7513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:35.259528 7513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.268360 7513 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:41089
I20260808 04:53:35.268424 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:41089 every 8 connection(s)
I20260808 04:53:35.269294 7513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:53:35.272610 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 7513
I20260808 04:53:35.272754 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:53:35.273036 15481 external_mini_cluster.cc:1428] Setting key ed9ae8dc61ec441c04a954b572acb7f0
Aug 08 04:53:35 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164815, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:53:35.284056 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.271145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:39055 (local address 127.15.30.126:44831)
0808 04:53:35.271304 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.271309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.272228 (+ 919us) server_negotiation.cc:407] Connection header received
0808 04:53:35.273701 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.273704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.273762 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.273845 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:35.275153 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.275957 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.277202 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.277438 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.277557 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.280449 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:35.280453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:35.280456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:35.280484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:35.281985 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.282524 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.282532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.282533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.282610 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.282984 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.282987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.282989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.283173 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:35.283282 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.283759 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.283885 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:53:35.284945 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.271588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:39055)
0808 04:53:35.272066 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.272113 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.273506 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.274044 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.274053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.274504 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:35.274978 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.274993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.276087 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.276089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.277061 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.277071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.277594 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.277598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.277816 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.278525 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:35.278541 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:35.280261 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:35.282120 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:35.282125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:35.282146 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:35.282407 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:35.282725 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:35.282728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:35.282730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:35.282879 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:35.283282 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:35.283288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:35.283516 (+ 228us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.283697 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.284000 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260808 04:53:35.286098 7723 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44831
I20260808 04:53:35.286440 7723 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
I20260808 04:53:35.287047 7723 heartbeater.cc:511] Master 127.15.30.126:44831 requested a full tablet report, sending...
I20260808 04:53:35.288134 7207 ts_manager.cc:194] Registered new tserver with Master: bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
I20260808 04:53:35.288780 7207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:39055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:53:35.301483 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44831
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:35.414080 7738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:35.414314 7738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:35.414381 7738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:35.418448 7738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:35.418504 7738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:35.418589 7738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:53:35.423741 7738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:35.424767 7738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:35.425776 7738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:35.436220 7749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:35.436208 7744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:35.436422 7738 server_base.cc:1034] running on GCE node
W20260808 04:53:35.436604 7746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:35.437086 7738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:35.437764 7738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:35.438931 7738 hybrid_clock.cc:648] HybridClock initialized: now 1786164815438897 us; error 49 us; skew 500 ppm
Aug 08 04:53:35 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164815, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:35.441810 7738 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:53:35.442718 7738 webserver.cc:460] Webserver started at http://127.15.30.67:38499/ using document root <none> and password file <none>
I20260808 04:53:35.443323 7738 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:35.443423 7738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:35.443692 7738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:53:35.445912 7738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "0a39598989aa42659b91e203637b1332"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "a97e15bf66382ec23dcf3058dd84b377"
server_key_iv: "1b5daf5ce7b1e748ce2b864ab5eeea02"
server_key_version: "encryptionkey@0"
I20260808 04:53:35.446514 7738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "0a39598989aa42659b91e203637b1332"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "a97e15bf66382ec23dcf3058dd84b377"
server_key_iv: "1b5daf5ce7b1e748ce2b864ab5eeea02"
server_key_version: "encryptionkey@0"
I20260808 04:53:35.450469 7738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:53:35.452908 7758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.454252 7738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:53:35.454413 7738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "0a39598989aa42659b91e203637b1332"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "a97e15bf66382ec23dcf3058dd84b377"
server_key_iv: "1b5daf5ce7b1e748ce2b864ab5eeea02"
server_key_version: "encryptionkey@0"
I20260808 04:53:35.454538 7738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:35.503598 7738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:35.506929 7738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:35.507159 7738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:35.507817 7738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:35.508859 7738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:35.508936 7738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.509018 7738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:35.509095 7738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:35.519032 7738 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:41317
I20260808 04:53:35.519057 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:41317 every 8 connection(s)
I20260808 04:53:35.519968 7738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:53:35.528537 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 7738
I20260808 04:53:35.528687 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:53:35.529088 15481 external_mini_cluster.cc:1428] Setting key 83543f954c1204e817e51a72f7ae995d
Aug 08 04:53:35 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164815, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:35.534161 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.521837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:58063 (local address 127.15.30.126:44831)
0808 04:53:35.522013 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.522019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.522881 (+ 862us) server_negotiation.cc:407] Connection header received
0808 04:53:35.524161 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.524165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.524214 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.524297 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:35.525476 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.526048 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.527145 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.527374 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.527461 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.530589 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:35.530593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:35.530596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:35.530627 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:35.532212 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.532744 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.532747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.532748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.532833 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.533156 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.533159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.533160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.533353 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:35.533436 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.533907 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.534010 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:53:35.534958 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.522259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:58063)
0808 04:53:35.522721 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.522764 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.523956 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.524473 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.524479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.524828 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:35.525337 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.525351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.526165 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.526168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.527025 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.527033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.527507 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.527509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.527644 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.528256 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:35.528281 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:35.530347 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:35.532324 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:35.532329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:35.532340 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:35.532646 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:35.532920 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:35.532923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:35.532924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:35.533054 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:35.533444 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:35.533449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:35.533663 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.533813 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.534053 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":133,"threads_started":1}
I20260808 04:53:35.536170 7953 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44831
I20260808 04:53:35.536461 7953 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:35.536929 7953 heartbeater.cc:511] Master 127.15.30.126:44831 requested a full tablet report, sending...
I20260808 04:53:35.537992 7207 ts_manager.cc:194] Registered new tserver with Master: 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317)
I20260808 04:53:35.538633 7207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:58063
I20260808 04:53:35.543026 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:53:35.552645 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.545129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.15.30.126:44831)
0808 04:53:35.545257 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.545260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.545522 (+ 262us) server_negotiation.cc:407] Connection header received
0808 04:53:35.545637 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.545640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.545674 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.545776 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:35.546515 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.546985 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.547819 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.548014 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.548085 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.549380 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:35.549382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:35.549384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:35.549405 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:35.550873 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.551355 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.551359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.551360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.551434 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:35.551734 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:35.551737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:35.551739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:35.551948 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:35.552090 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.552338 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.552487 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:53:35.555219 7207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:53:35.558403 7207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:53:35.573086 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.567201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:43533 (local address 127.0.0.1:60574)
0808 04:53:35.568019 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.568032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.568129 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.568889 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.568892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.568925 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:35.569118 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.569127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.570565 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.570568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.571716 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.571722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.572628 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.572632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.572744 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.572772 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.572843 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.572913 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":31,"spinlock_wait_cycles":768,"thread_start_us":106,"threads_started":1}
I20260808 04:53:35.573093 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.567658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.15.30.67:41317)
0808 04:53:35.568108 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.568117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.568335 (+ 218us) server_negotiation.cc:407] Connection header received
0808 04:53:35.568513 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.568519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.568729 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.568883 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:35.569227 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.569997 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.571484 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.572226 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.572338 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.572509 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.572847 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.572942 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":183,"threads_started":1}
I20260808 04:53:35.573356 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.566737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41089 (local address 127.0.0.1:60816)
0808 04:53:35.567103 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.567156 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.567298 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.568529 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.568534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.568557 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:35.568710 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.568718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.570264 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.570267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.571696 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.571708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.572989 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.572995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.573077 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.573098 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.573168 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.573229 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":57,"thread_start_us":95,"threads_started":1}
I20260808 04:53:35.572971 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.567508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41317 (local address 127.0.0.1:36304)
0808 04:53:35.568230 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.568240 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.568357 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.568859 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.568861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.568878 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:35.569104 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.569111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.570151 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.570154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.571359 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.571369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.572389 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.572396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.572519 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.572577 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.572685 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.572817 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":165,"spinlock_wait_cycles":2944,"thread_start_us":90,"threads_started":1}
I20260808 04:53:35.573887 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.567649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.15.30.65:43533)
0808 04:53:35.568407 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.568413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.568435 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:53:35.568513 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.568521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.568710 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.569282 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:35.569413 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.570442 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.571849 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.572455 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.572596 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.572753 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.573045 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.573146 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":99,"threads_started":1}
I20260808 04:53:35.573962 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.566892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.15.30.66:41089)
0808 04:53:35.567885 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.567898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.567928 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:53:35.568013 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.568019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.568258 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.568458 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:35.568858 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.570110 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.571831 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.572664 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.572778 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.572927 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.573698 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.573801 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":126,"threads_started":1}
I20260808 04:53:35.576327 7838 tablet_service.cc:1467] Processing CreateTablet for tablet 5b079e21da89455f99ef83d0bc82c632 (DEFAULT_TABLE table=test-table [id=855bef370cae49e6b282f22e66e68f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:35.576730 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 5b079e21da89455f99ef83d0bc82c632 (DEFAULT_TABLE table=test-table [id=855bef370cae49e6b282f22e66e68f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:35.576732 7619 tablet_service.cc:1467] Processing CreateTablet for tablet 5b079e21da89455f99ef83d0bc82c632 (DEFAULT_TABLE table=test-table [id=855bef370cae49e6b282f22e66e68f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:35.577594 7838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b079e21da89455f99ef83d0bc82c632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:35.577749 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b079e21da89455f99ef83d0bc82c632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:35.577749 7619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b079e21da89455f99ef83d0bc82c632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:35.583957 7982 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Bootstrap starting.
I20260808 04:53:35.585384 7985 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap starting.
I20260808 04:53:35.586252 7982 tablet_bootstrap.cc:654] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:35.587154 7985 tablet_bootstrap.cc:654] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:35.587466 7982 log.cc:824] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:35.588056 7985 log.cc:824] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:35.588424 7984 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Bootstrap starting.
I20260808 04:53:35.589956 7985 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: No bootstrap required, opened a new log
I20260808 04:53:35.589956 7982 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: No bootstrap required, opened a new log
I20260808 04:53:35.590168 7985 ts_tablet_manager.cc:1397] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 04:53:35.590178 7982 ts_tablet_manager.cc:1397] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:53:35.590493 7984 tablet_bootstrap.cc:654] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:35.591148 7984 log.cc:824] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:35.592872 7984 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: No bootstrap required, opened a new log
I20260808 04:53:35.593075 7984 ts_tablet_manager.cc:1397] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:53:35.594300 7982 raft_consensus.cc:348] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.594609 7982 raft_consensus.cc:374] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:35.594662 7982 raft_consensus.cc:729] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a39598989aa42659b91e203637b1332, State: Initialized, Role: FOLLOWER
I20260808 04:53:35.594839 7985 raft_consensus.cc:348] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.595094 7985 raft_consensus.cc:374] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:35.595162 7985 raft_consensus.cc:729] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08399aa6dda34b4ba17a77cbbb51bffc, State: Initialized, Role: FOLLOWER
I20260808 04:53:35.595122 7982 consensus_queue.cc:260] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.595628 7985 consensus_queue.cc:260] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.596005 7953 heartbeater.cc:503] Master 127.15.30.126:44831 was elected leader, sending a full tablet report...
I20260808 04:53:35.596160 7982 ts_tablet_manager.cc:1428] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:35.596406 7507 heartbeater.cc:503] Master 127.15.30.126:44831 was elected leader, sending a full tablet report...
I20260808 04:53:35.596587 7985 ts_tablet_manager.cc:1428] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent starting tablet: real 0.006s user 0.003s sys 0.005s
I20260808 04:53:35.597069 7984 raft_consensus.cc:348] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.597293 7984 raft_consensus.cc:374] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:35.597354 7984 raft_consensus.cc:729] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb869d8a50c4226ae0ccead02b414d5, State: Initialized, Role: FOLLOWER
I20260808 04:53:35.597790 7984 consensus_queue.cc:260] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.598500 7723 heartbeater.cc:503] Master 127.15.30.126:44831 was elected leader, sending a full tablet report...
I20260808 04:53:35.598701 7984 ts_tablet_manager.cc:1428] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:35.646108 7994 raft_consensus.cc:482] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:35.646360 7994 raft_consensus.cc:504] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.647544 7994 leader_election.cc:288] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089), 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317)
I20260808 04:53:35.652988 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.648358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:55735 (local address 127.15.30.67:41317)
0808 04:53:35.648518 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.648525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.648689 (+ 164us) server_negotiation.cc:407] Connection header received
0808 04:53:35.648785 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.648787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.648829 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.648916 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:35.649223 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.650005 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.651554 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.652261 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.652413 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.652468 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.652774 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.652849 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:53:35.652988 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.647892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:38065 (local address 127.15.30.66:41089)
0808 04:53:35.648041 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.648044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.648187 (+ 143us) server_negotiation.cc:407] Connection header received
0808 04:53:35.648412 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.648417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.648484 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.648594 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:35.648971 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.649876 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.651554 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.652243 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.652409 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.652468 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.652774 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.652849 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:53:35.653036 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.647749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41089 (local address 127.15.30.65:38065)
0808 04:53:35.648067 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.648084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.648205 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.648618 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.648624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.648652 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:35.648829 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.648835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.649990 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.649993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.651393 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.651406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.652405 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.652409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.652614 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.652659 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.652738 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.652845 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":186,"spinlock_wait_cycles":768,"thread_start_us":148,"threads_started":1}
I20260808 04:53:35.653036 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.648208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41317 (local address 127.15.30.65:55735)
0808 04:53:35.648578 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.648591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:35.648693 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:35.648939 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:35.648944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:35.648962 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:35.649064 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.649068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.650114 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.650117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.651393 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:35.651406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.652408 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.652411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:35.652602 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.652654 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 04:53:35.652738 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:53:35.652841 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":78,"thread_start_us":107,"threads_started":1}
I20260808 04:53:35.653786 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b079e21da89455f99ef83d0bc82c632" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39598989aa42659b91e203637b1332" is_pre_election: true
I20260808 04:53:35.653786 7647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b079e21da89455f99ef83d0bc82c632" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb869d8a50c4226ae0ccead02b414d5" is_pre_election: true
I20260808 04:53:35.654086 7876 raft_consensus.cc:2454] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 0.
I20260808 04:53:35.654121 7647 raft_consensus.cc:2454] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 0.
I20260808 04:53:35.654574 7326 leader_election.cc:302] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc, 0a39598989aa42659b91e203637b1332; no voters:
I20260808 04:53:35.654878 7994 raft_consensus.cc:2781] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:53:35.654966 7994 raft_consensus.cc:482] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:53:35.655000 7994 raft_consensus.cc:3037] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:35.656157 7994 raft_consensus.cc:504] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.656556 7994 leader_election.cc:288] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 election: Requested vote from peers bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089), 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317)
I20260808 04:53:35.657023 7647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b079e21da89455f99ef83d0bc82c632" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb869d8a50c4226ae0ccead02b414d5"
I20260808 04:53:35.657061 7876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b079e21da89455f99ef83d0bc82c632" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39598989aa42659b91e203637b1332"
I20260808 04:53:35.657137 7647 raft_consensus.cc:3037] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:35.657214 7876 raft_consensus.cc:3037] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:35.658352 7647 raft_consensus.cc:2454] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 1.
I20260808 04:53:35.658425 7876 raft_consensus.cc:2454] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 1.
I20260808 04:53:35.658754 7326 leader_election.cc:302] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc, bfb869d8a50c4226ae0ccead02b414d5; no voters:
I20260808 04:53:35.658946 7994 raft_consensus.cc:2781] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:35.659179 7994 raft_consensus.cc:686] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 LEADER]: Becoming Leader. State: Replica: 08399aa6dda34b4ba17a77cbbb51bffc, State: Running, Role: LEADER
I20260808 04:53:35.659513 7994 consensus_queue.cc:237] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:35.662834 7206 catalog_manager.cc:5495] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc reported cstate change: term changed from 0 to 1, leader changed from <none> to 08399aa6dda34b4ba17a77cbbb51bffc (127.15.30.65). New cstate: current_term: 1 leader_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260808 04:53:35.694809 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.691334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.15.30.65:43533)
0808 04:53:35.691463 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:35.691467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:35.691480 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:53:35.691553 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:35.691556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:35.691617 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:35.691707 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:53:35.691944 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.692502 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.693333 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:35.693553 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:35.693666 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:35.693936 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:53:35.694378 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:53:35.694476 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:35.694533 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 04:53:35.694585 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:53:35.700788 7876 raft_consensus.cc:1261] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Refusing update from remote peer 08399aa6dda34b4ba17a77cbbb51bffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:35.700794 7647 raft_consensus.cc:1261] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Refusing update from remote peer 08399aa6dda34b4ba17a77cbbb51bffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:35.701486 8001 consensus_queue.cc:1035] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:35.701655 7994 consensus_queue.cc:1035] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:35.710788 8009 mvcc.cc:205] Tried to move back new op lower bound from 7316131085102903296 to 7316131084943974400. Current Snapshot: MvccSnapshot[applied={T|T < 7316131085102903296}]
I20260808 04:53:35.720057 7206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:35.720250 7206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:35.722857 7206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Sending DeleteTablet for 3 replicas of tablet 5b079e21da89455f99ef83d0bc82c632
I20260808 04:53:35.723755 7410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b079e21da89455f99ef83d0bc82c632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:60574
I20260808 04:53:35.723779 7838 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b079e21da89455f99ef83d0bc82c632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:36304
I20260808 04:53:35.724319 8030 tablet_replica.cc:330] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: stopping tablet replica
I20260808 04:53:35.724303 7619 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b079e21da89455f99ef83d0bc82c632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:60816
I20260808 04:53:35.724588 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 7299
I20260808 04:53:35.738679 8032 tablet_replica.cc:330] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: stopping tablet replica
I20260808 04:53:35.739064 8032 raft_consensus.cc:2229] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:35.739444 8032 raft_consensus.cc:2258] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:35.741879 8032 ts_tablet_manager.cc:1905] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:35.724699 8030 raft_consensus.cc:2229] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:35.742961 8030 raft_consensus.cc:2258] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:35.744661 8030 ts_tablet_manager.cc:1905] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:35.745493 8032 ts_tablet_manager.cc:1918] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:35.745604 8032 log.cc:1196] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/5b079e21da89455f99ef83d0bc82c632
I20260808 04:53:35.745966 8032 ts_tablet_manager.cc:1939] T 5b079e21da89455f99ef83d0bc82c632 P bfb869d8a50c4226ae0ccead02b414d5: Deleting consensus metadata
I20260808 04:53:35.747432 7180 catalog_manager.cc:4841] TS bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089): tablet 5b079e21da89455f99ef83d0bc82c632 (table test-table [id=855bef370cae49e6b282f22e66e68f87]) successfully deleted
I20260808 04:53:35.747920 8030 ts_tablet_manager.cc:1918] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:35.748005 8030 log.cc:1196] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/5b079e21da89455f99ef83d0bc82c632
I20260808 04:53:35.748361 8030 ts_tablet_manager.cc:1939] T 5b079e21da89455f99ef83d0bc82c632 P 0a39598989aa42659b91e203637b1332: Deleting consensus metadata
I20260808 04:53:35.749600 7180 catalog_manager.cc:4841] TS 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317): tablet 5b079e21da89455f99ef83d0bc82c632 (table test-table [id=855bef370cae49e6b282f22e66e68f87]) successfully deleted
I20260808 04:53:35.749693 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 7513
W20260808 04:53:35.749739 7187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:43533 (error 108)
I20260808 04:53:35.755347 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:35.754975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:43533 (local address 127.0.0.1:60588)
0808 04:53:35.755157 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:35.755242 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 04:53:35.755540 7187 catalog_manager.cc:4584] TS 08399aa6dda34b4ba17a77cbbb51bffc (127.15.30.65:43533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b079e21da89455f99ef83d0bc82c632: Network error: Client connection negotiation failed: client connection to 127.15.30.65:43533: connect: Connection refused (error 111)
I20260808 04:53:35.759665 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 7738
I20260808 04:53:35.776039 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 7159
I20260808 04:53:55.787616 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44831
--webserver_interface=127.15.30.126
--webserver_port=44835
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:53:55.905678 8034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:55.905941 8034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:55.905982 8034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:55.909938 8034 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:53:55.909986 8034 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:53:55.910001 8034 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:53:55.910015 8034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:53:55.910053 8034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:55.910092 8034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:55.910120 8034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:53:55.910157 8034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:53:55.915458 8034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:55.916517 8034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:55.917546 8034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:55.931733 8048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:55.931707 8034 server_base.cc:1034] running on GCE node
W20260808 04:53:55.931679 8044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:55.931753 8043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:55.932621 8034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:55.933766 8034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:55.934957 8034 hybrid_clock.cc:648] HybridClock initialized: now 1786164835934942 us; error 30 us; skew 500 ppm
Aug 08 04:53:55 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164835, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:55.937903 8034 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:53:55.938817 8034 webserver.cc:460] Webserver started at http://127.15.30.126:44835/ using document root <none> and password file <none>
I20260808 04:53:55.939421 8034 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:55.939534 8034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:55.943567 8034 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:53:55.945878 8061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:55.947130 8034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:53:55.947295 8034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "846de0155ad1466184e63b32f1268d18"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "35c0436360f4d849d82cee04218c000d"
server_key_iv: "7624b4e5efe178cf3ea05511461c70d3"
server_key_version: "encryptionkey@0"
I20260808 04:53:55.947883 8034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:55.964056 8034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:55.967130 8034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:55.967352 8034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:55.974838 8034 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:44831
I20260808 04:53:55.975157 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:44831 every 8 connection(s)
I20260808 04:53:55.975919 8034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:53:55.976218 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 8034
I20260808 04:53:55.976785 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:43533
--local_ip_for_outbound_sockets=127.15.30.65
--tserver_master_addrs=127.15.30.126:44831
--webserver_port=37769
--webserver_interface=127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:53:55.980669 8143 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:53:55.982354 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Bootstrap starting.
I20260808 04:53:56.000612 8143 log.cc:824] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:56.009500 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:53:56.009909 8143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Bootstrap complete.
I20260808 04:53:56.014213 8143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:56.014631 8143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846de0155ad1466184e63b32f1268d18, State: Initialized, Role: FOLLOWER
I20260808 04:53:56.015120 8143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:56.015269 8143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:53:56.015362 8143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:53:56.015475 8143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:53:56.017460 8143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:56.017807 8143 leader_election.cc:302] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846de0155ad1466184e63b32f1268d18; no voters:
I20260808 04:53:56.018141 8143 leader_election.cc:288] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:53:56.018236 8151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:53:56.018433 8151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 LEADER]: Becoming Leader. State: Replica: 846de0155ad1466184e63b32f1268d18, State: Running, Role: LEADER
I20260808 04:53:56.018787 8151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:53:56.019433 8143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:53:56.020270 8152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "846de0155ad1466184e63b32f1268d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } } }
I20260808 04:53:56.020431 8152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:56.021013 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846de0155ad1466184e63b32f1268d18. Latest consensus state: current_term: 2 leader_uuid: "846de0155ad1466184e63b32f1268d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } } }
I20260808 04:53:56.021159 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: This master's current role is: LEADER
I20260808 04:53:56.021427 8163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:53:56.026571 8163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:53:56.027169 8163 catalog_manager.cc:1259] Loaded cluster ID: c9f6e7f77f584daea01459f088ff4985
I20260808 04:53:56.027235 8163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:53:56.030386 8163 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:53:56.031069 8163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Loaded TSK: 0
I20260808 04:53:56.044198 8163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Generated new TSK 1
I20260808 04:53:56.044611 8163 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:53:56.099459 8147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:56.099709 8147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:56.099751 8147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:56.103845 8147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:56.103906 8147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:56.103965 8147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:53:56.109622 8147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:43533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=37769
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:56.110626 8147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:56.111614 8147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:56.123247 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:56.123260 8188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:56.123641 8147 server_base.cc:1034] running on GCE node
W20260808 04:53:56.123268 8190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:56.124274 8147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:56.124986 8147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:56.126171 8147 hybrid_clock.cc:648] HybridClock initialized: now 1786164836126157 us; error 73 us; skew 500 ppm
Aug 08 04:53:56 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:56.129017 8147 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:53:56.129976 8147 webserver.cc:460] Webserver started at http://127.15.30.65:37769/ using document root <none> and password file <none>
I20260808 04:53:56.130539 8147 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:56.130615 8147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:56.134568 8147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:53:56.136977 8212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:56.138244 8147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:53:56.138373 8147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "08399aa6dda34b4ba17a77cbbb51bffc"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "a223d8a89fe3c2cd16260ab0340eadd8"
server_key_iv: "c2eb83af2494bfdd6283a3896fe36a97"
server_key_version: "encryptionkey@0"
I20260808 04:53:56.138927 8147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:56.149462 8147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:56.153152 8147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:56.153369 8147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:56.154135 8147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:56.155435 8223 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:53:56.160979 8147 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:53:56.161068 8147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260808 04:53:56.161158 8147 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:53:56.162943 8147 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:53:56.163014 8147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:53:56.163340 8223 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap starting.
I20260808 04:53:56.174108 8147 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:43533
I20260808 04:53:56.174203 8399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:43533 every 8 connection(s)
I20260808 04:53:56.175096 8147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:53:56.183472 8223 log.cc:824] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:56.184610 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 8147
I20260808 04:53:56.185182 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:41089
--local_ip_for_outbound_sockets=127.15.30.66
--tserver_master_addrs=127.15.30.126:44831
--webserver_port=34177
--webserver_interface=127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:53:56 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:56.191917 8223 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:53:56.192361 8223 tablet_bootstrap.cc:492] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap complete.
I20260808 04:53:56.193755 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.177416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:33625 (local address 127.15.30.126:44831)
0808 04:53:56.177860 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.177886 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.178225 (+ 339us) server_negotiation.cc:407] Connection header received
0808 04:53:56.179354 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.179375 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.179887 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.180288 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:56.181416 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.182918 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.184207 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.184874 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.185063 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.187656 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:56.187663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:56.187683 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:56.187725 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:56.189821 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.190481 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.190487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.190493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.190651 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.191098 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.191101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.191103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.191564 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:56.191721 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.192519 (+ 798us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.192856 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":152,"threads_started":1}
I20260808 04:53:56.193799 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.177369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:33625)
0808 04:53:56.178023 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.178078 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.178897 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.180334 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.180341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.180759 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:56.181222 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.181236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.183076 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.183079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.184073 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.184083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.185031 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.185035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.185238 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.185937 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:56.185954 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:56.187445 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:56.189953 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.189962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.189979 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.190345 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.190781 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.190787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.190792 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.190987 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.191708 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:56.191726 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:56.192056 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.192288 (+ 232us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.192691 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":139,"threads_started":1}
I20260808 04:53:56.193813 8223 ts_tablet_manager.cc:1397] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.009s
I20260808 04:53:56.196559 8400 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44831
I20260808 04:53:56.196841 8400 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:56.196899 8223 raft_consensus.cc:348] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.197361 8223 raft_consensus.cc:729] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08399aa6dda34b4ba17a77cbbb51bffc, State: Initialized, Role: FOLLOWER
I20260808 04:53:56.197459 8400 heartbeater.cc:511] Master 127.15.30.126:44831 requested a full tablet report, sending...
I20260808 04:53:56.197829 8223 consensus_queue.cc:260] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.198817 8223 ts_tablet_manager.cc:1428] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:53:56.200636 8092 ts_manager.cc:194] Registered new tserver with Master: 08399aa6dda34b4ba17a77cbbb51bffc (127.15.30.65:43533)
I20260808 04:53:56.204659 8092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:33625
I20260808 04:53:56.207193 8400 heartbeater.cc:503] Master 127.15.30.126:44831 was elected leader, sending a full tablet report...
Aug 08 04:53:56 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164835, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.65@KRBTEST.COM
I20260808 04:53:56.215080 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.203625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:43533 (local address 127.0.0.1:42388)
0808 04:53:56.204077 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.204128 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.204261 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.204923 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.204930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.205414 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:56.205574 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.205583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.207064 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.207068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.208006 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.208015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.208858 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.208862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.208955 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.209517 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:56.209546 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:56.211219 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:56.213473 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.213482 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.213486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.213727 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.214048 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.214052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.214054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.214139 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.214615 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:56.214620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:56.214714 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.214819 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.214954 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260808 04:53:56.215236 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.203729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.15.30.65:43533)
0808 04:53:56.204346 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.204356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.204413 (+ 57us) server_negotiation.cc:407] Connection header received
0808 04:53:56.204481 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.204486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.204670 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.204807 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:56.205729 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.206925 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.208134 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.208617 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.208758 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.211366 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:56.211376 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:56.211390 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:56.211417 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:56.213364 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.213837 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.213843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.213845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.213947 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.214225 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.214228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.214229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.214503 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:56.214622 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.214919 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.215082 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":147,"threads_started":1}
I20260808 04:53:56.216140 8294 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b079e21da89455f99ef83d0bc82c632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42388
I20260808 04:53:56.216979 8417 tablet_replica.cc:330] stopping tablet replica
I20260808 04:53:56.217267 8417 raft_consensus.cc:2229] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:56.217499 8417 raft_consensus.cc:2258] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:56.218449 8417 ts_tablet_manager.cc:1905] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:56.222500 8417 ts_tablet_manager.cc:1918] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:56.222641 8417 log.cc:1196] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal/wals/5b079e21da89455f99ef83d0bc82c632
I20260808 04:53:56.222956 8417 ts_tablet_manager.cc:1939] T 5b079e21da89455f99ef83d0bc82c632 P 08399aa6dda34b4ba17a77cbbb51bffc: Deleting consensus metadata
I20260808 04:53:56.224633 8067 catalog_manager.cc:4841] TS 08399aa6dda34b4ba17a77cbbb51bffc (127.15.30.65:43533): tablet 5b079e21da89455f99ef83d0bc82c632 (table test-table [id=855bef370cae49e6b282f22e66e68f87]) successfully deleted
W20260808 04:53:56.305593 8409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:56.305825 8409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:56.305864 8409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:56.309857 8409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:56.309911 8409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:56.310005 8409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:53:56.315054 8409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:41089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=34177
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:56.315977 8409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:56.316994 8409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:56.326509 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:56.326659 8409 server_base.cc:1034] running on GCE node
W20260808 04:53:56.326527 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:56.326548 8432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:56.327119 8409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:56.327733 8409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:56.328945 8409 hybrid_clock.cc:648] HybridClock initialized: now 1786164836328924 us; error 42 us; skew 500 ppm
Aug 08 04:53:56 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:56.331544 8409 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:53:56.332342 8409 webserver.cc:460] Webserver started at http://127.15.30.66:34177/ using document root <none> and password file <none>
I20260808 04:53:56.332896 8409 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:56.332969 8409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:56.336609 8409 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 04:53:56.338541 8442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:56.339768 8409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:53:56.339893 8409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "bfb869d8a50c4226ae0ccead02b414d5"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "c7b0c2f64bc66e362e837e9f58869dda"
server_key_iv: "ba6e956b28408ff4c4b5ccdfa6ec95e5"
server_key_version: "encryptionkey@0"
I20260808 04:53:56.340476 8409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:56.363633 8409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:56.366938 8409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:56.367091 8409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:56.367601 8409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:56.368438 8409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:56.368489 8409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:56.368530 8409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:56.368544 8409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:56.377089 8409 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:41089
I20260808 04:53:56.377151 8615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:41089 every 8 connection(s)
I20260808 04:53:56.378062 8409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:53:56.384765 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 8409
I20260808 04:53:56.385229 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:41317
--local_ip_for_outbound_sockets=127.15.30.67
--tserver_master_addrs=127.15.30.126:44831
--webserver_port=38499
--webserver_interface=127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:53:56 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:56.391229 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.379826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:49911 (local address 127.15.30.126:44831)
0808 04:53:56.379968 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.379972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.380811 (+ 839us) server_negotiation.cc:407] Connection header received
0808 04:53:56.382078 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.382080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.382127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.382200 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:56.383290 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.383802 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.384987 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.385228 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.385342 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.387650 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:56.387659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:56.387661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:56.387685 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:56.389201 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.389713 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.389717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.389719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.389799 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.390132 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.390135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.390137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.390312 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:56.390417 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.390883 (+ 466us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.391020 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:53:56.391943 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.380183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:49911)
0808 04:53:56.380670 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.380711 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.381892 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.382361 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.382367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.382720 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:56.383163 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.383176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.383909 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.383912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.384851 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.384865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.385380 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.385385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.385551 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.386089 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:56.386105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:56.387487 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:56.389330 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.389335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.389349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.389604 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.389892 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.389895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.389897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.390032 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.390416 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:56.390423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:56.390659 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.390840 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.391138 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":163,"threads_started":1}
I20260808 04:53:56.393103 8617 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44831
I20260808 04:53:56.393373 8617 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:56.394016 8617 heartbeater.cc:511] Master 127.15.30.126:44831 requested a full tablet report, sending...
I20260808 04:53:56.395210 8092 ts_manager.cc:194] Registered new tserver with Master: bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
I20260808 04:53:56.396098 8092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:49911
W20260808 04:53:56.507726 8622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:53:56.508025 8622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:53:56.508083 8622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:53:56.512509 8622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:53:56.512564 8622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:53:56.512625 8622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:53:56.518263 8622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:41317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=38499
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:53:56.519299 8622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:53:56.520335 8622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:53:56.530444 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:56.530658 8622 server_base.cc:1034] running on GCE node
W20260808 04:53:56.530444 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:53:56.530835 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:53:56.531270 8622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:53:56.531865 8622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:53:56.533030 8622 hybrid_clock.cc:648] HybridClock initialized: now 1786164836533015 us; error 28 us; skew 500 ppm
Aug 08 04:53:56 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:53:56.535887 8622 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:53:56.536834 8622 webserver.cc:460] Webserver started at http://127.15.30.67:38499/ using document root <none> and password file <none>
I20260808 04:53:56.537434 8622 fs_manager.cc:359] Metadata directory not provided
I20260808 04:53:56.537549 8622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:53:56.541530 8622 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:53:56.543815 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:56.545238 8622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:53:56.545370 8622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "0a39598989aa42659b91e203637b1332"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "a97e15bf66382ec23dcf3058dd84b377"
server_key_iv: "1b5daf5ce7b1e748ce2b864ab5eeea02"
server_key_version: "encryptionkey@0"
I20260808 04:53:56.545904 8622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:53:56.557578 8622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:53:56.560635 8622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:53:56.560818 8622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:53:56.561359 8622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:53:56.562125 8622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:53:56.562212 8622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:56.562294 8622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:53:56.562340 8622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:53:56.570850 8622 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:41317
I20260808 04:53:56.570917 8824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:41317 every 8 connection(s)
I20260808 04:53:56.571794 8622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:53:56.572728 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 8622
I20260808 04:53:56.581832 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.573490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.15.30.126:44831)
0808 04:53:56.573641 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.573645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.573899 (+ 254us) server_negotiation.cc:407] Connection header received
0808 04:53:56.574025 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.574028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.574070 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.574141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:56.575155 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.576017 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.576925 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.577275 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.577393 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.578677 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:56.578681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:56.578683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:56.578709 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:56.580113 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.580631 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.580635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.580637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.580723 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.580995 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.580999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.581001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.581180 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:56.581400 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.581545 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.581662 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 08 04:53:56 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164836, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:53:56.586333 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.573743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:53573 (local address 127.15.30.126:44831)
0808 04:53:56.573962 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.573965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.574941 (+ 976us) server_negotiation.cc:407] Connection header received
0808 04:53:56.576610 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.576614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.576665 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.576752 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:56.578038 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.578549 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.579931 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.580172 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.580278 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.582809 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:56.582812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:56.582813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:56.582860 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:56.584616 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.585069 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.585072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.585073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.585172 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.585464 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.585466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.585467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.585614 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:56.585689 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.586107 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.586219 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260808 04:53:56.587116 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.574248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:53573)
0808 04:53:56.574798 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.574837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.576415 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.576942 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.576950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.577415 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:53:56.577896 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.577909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.578678 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.578682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.579807 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.579818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.580320 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.580324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.580529 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.581043 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:53:56.581058 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:53:56.582650 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:53:56.584722 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.584726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.584736 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.584984 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.585257 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:53:56.585259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:53:56.585261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:53:56.585386 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:53:56.585697 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:53:56.585702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:53:56.585916 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.586062 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.586289 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":169,"threads_started":1}
I20260808 04:53:56.588265 8828 heartbeater.cc:347] Connected to a master server at 127.15.30.126:44831
I20260808 04:53:56.588510 8828 heartbeater.cc:464] Registering TS with master...
I20260808 04:53:56.588976 8828 heartbeater.cc:511] Master 127.15.30.126:44831 requested a full tablet report, sending...
I20260808 04:53:56.590049 8092 ts_manager.cc:194] Registered new tserver with Master: 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317)
I20260808 04:53:56.590602 8092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:53573
I20260808 04:53:56.595803 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:53:56.599541 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.596440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.15.30.126:44831)
0808 04:53:56.596559 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.596562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.596764 (+ 202us) server_negotiation.cc:407] Connection header received
0808 04:53:56.596890 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.596892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.596978 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.597062 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:53:56.597285 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.597761 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.598596 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.598807 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.598881 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.599130 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:53:56.599266 (+ 136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 04:53:56.599452 (+ 186us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58550: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 04:53:56.599624 8831 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58550: token expired
W20260808 04:53:56.599959 7958 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.30.126:44831)
I20260808 04:53:56.607067 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.600293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.15.30.126:44831)
0808 04:53:56.600416 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.600421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.600459 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:53:56.600541 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.600542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.600573 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.600633 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:53:56.601257 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.601760 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.602544 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.602763 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.602844 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.604135 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:53:56.604138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:53:56.604140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:53:56.604163 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:53:56.605523 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.605943 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.605946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.605947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.606058 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:53:56.606343 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:53:56.606345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:53:56.606346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:53:56.606500 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:53:56.606604 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.606857 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.606967 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 04:53:56.609385 8092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:53:56.612640 8092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:53:56.620325 8294 tablet_service.cc:1467] Processing CreateTablet for tablet f5b04ff7675e41ab997ca66cd7af7fd4 (DEFAULT_TABLE table=test-table [id=567ebf50d546456c8945486eb597e112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:56.621060 8294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b04ff7675e41ab997ca66cd7af7fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:56.623838 8223 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap starting.
I20260808 04:53:56.624713 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.619253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41317 (local address 127.0.0.1:38992)
0808 04:53:56.619364 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.619381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.619462 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.620429 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.620432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.620453 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:56.620573 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.620583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.622149 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.622152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.623100 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.623109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.624185 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.624189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.624315 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.624359 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.624455 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.624543 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 04:53:56.624969 8223 tablet_bootstrap.cc:654] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:56.625361 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.619463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.15.30.67:41317)
0808 04:53:56.619886 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.619899 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.619924 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:53:56.620002 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.620008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.620221 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.620467 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:56.620848 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.622008 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.623232 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.624048 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.624913 (+ 865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.625076 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.625135 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.625248 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":176,"threads_started":1}
I20260808 04:53:56.625941 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.619529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41089 (local address 127.0.0.1:49446)
0808 04:53:56.620437 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.620450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.620537 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.621369 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.621372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.621392 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:56.621478 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.621484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.622667 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.622670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.624064 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.624075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.625505 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.625511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.625630 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.625657 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.625755 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.625824 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":126,"threads_started":1}
I20260808 04:53:56.626070 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.620007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.15.30.66:41089)
0808 04:53:56.620319 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.620332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.620658 (+ 326us) server_negotiation.cc:407] Connection header received
0808 04:53:56.620786 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.620807 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.621022 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.621188 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:56.621581 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.622536 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.624243 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.625157 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.625249 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.625395 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.625793 (+ 398us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.625898 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":129,"threads_started":1}
I20260808 04:53:56.626540 8223 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: No bootstrap required, opened a new log
I20260808 04:53:56.626649 8223 ts_tablet_manager.cc:1397] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 04:53:56.627127 8223 raft_consensus.cc:348] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.627254 8223 raft_consensus.cc:374] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:56.627295 8223 raft_consensus.cc:729] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08399aa6dda34b4ba17a77cbbb51bffc, State: Initialized, Role: FOLLOWER
I20260808 04:53:56.627624 8223 consensus_queue.cc:260] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.627914 8223 ts_tablet_manager.cc:1428] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:53:56.628747 8725 tablet_service.cc:1467] Processing CreateTablet for tablet f5b04ff7675e41ab997ca66cd7af7fd4 (DEFAULT_TABLE table=test-table [id=567ebf50d546456c8945486eb597e112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:56.629138 8517 tablet_service.cc:1467] Processing CreateTablet for tablet f5b04ff7675e41ab997ca66cd7af7fd4 (DEFAULT_TABLE table=test-table [id=567ebf50d546456c8945486eb597e112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:53:56.629920 8725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b04ff7675e41ab997ca66cd7af7fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:56.630209 8517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b04ff7675e41ab997ca66cd7af7fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:53:56.636732 8843 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Bootstrap starting.
I20260808 04:53:56.637615 8844 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Bootstrap starting.
I20260808 04:53:56.639109 8843 tablet_bootstrap.cc:654] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:56.640249 8844 tablet_bootstrap.cc:654] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Neither blocks nor log segments found. Creating new log.
I20260808 04:53:56.640576 8843 log.cc:824] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:56.641115 8844 log.cc:824] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:53:56.642709 8843 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: No bootstrap required, opened a new log
I20260808 04:53:56.642908 8843 ts_tablet_manager.cc:1397] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:56.643059 8844 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: No bootstrap required, opened a new log
I20260808 04:53:56.643263 8844 ts_tablet_manager.cc:1397] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:53:56.646185 8843 raft_consensus.cc:348] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.646391 8843 raft_consensus.cc:374] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:56.646241 8844 raft_consensus.cc:348] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.646446 8843 raft_consensus.cc:729] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a39598989aa42659b91e203637b1332, State: Initialized, Role: FOLLOWER
I20260808 04:53:56.646479 8844 raft_consensus.cc:374] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:53:56.646539 8844 raft_consensus.cc:729] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb869d8a50c4226ae0ccead02b414d5, State: Initialized, Role: FOLLOWER
I20260808 04:53:56.646883 8843 consensus_queue.cc:260] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.646903 8844 consensus_queue.cc:260] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.647465 8828 heartbeater.cc:503] Master 127.15.30.126:44831 was elected leader, sending a full tablet report...
I20260808 04:53:56.647792 8844 ts_tablet_manager.cc:1428] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 04:53:56.648319 8617 heartbeater.cc:503] Master 127.15.30.126:44831 was elected leader, sending a full tablet report...
I20260808 04:53:56.649967 8843 ts_tablet_manager.cc:1428] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:53:56.670974 8410 raft_consensus.cc:482] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:53:56.671198 8410 raft_consensus.cc:504] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.672292 8410 leader_election.cc:288] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317), bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
W20260808 04:53:56.676276 8402 tablet.cc:2369] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:53:56.677433 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.672667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35621 (local address 127.15.30.67:41317)
0808 04:53:56.672799 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.672816 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.672835 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:53:56.673118 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.673124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.673174 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.673282 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:56.673600 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.674371 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.676064 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.676697 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.676831 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.676881 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.677223 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.677312 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:53:56.677444 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.672990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:45861 (local address 127.15.30.66:41089)
0808 04:53:56.673110 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.673114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.673255 (+ 141us) server_negotiation.cc:407] Connection header received
0808 04:53:56.673376 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.673379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.673426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.673486 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:53:56.673763 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.674483 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.676064 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.676714 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.676831 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.676881 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.677281 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.677345 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:53:56.677452 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.672568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41317 (local address 127.15.30.65:35621)
0808 04:53:56.672697 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.672715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.672821 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.673308 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.673313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.673334 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:56.673463 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.673486 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.674504 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.674507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.675928 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.675941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.676834 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.676840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.677021 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.677102 (+ 81us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.677208 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.677296 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":83}
I20260808 04:53:56.677480 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.672863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41089 (local address 127.15.30.65:45861)
0808 04:53:56.673158 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.673170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:53:56.673288 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:53:56.673520 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:53:56.673526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:53:56.673546 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:53:56.673661 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.673666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.674590 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.674593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.675928 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:53:56.675941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.676848 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.676851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:53:56.677004 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.677172 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 04:53:56.677242 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 04:53:56.677295 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":196,"thread_start_us":94,"threads_started":1}
I20260808 04:53:56.678385 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b04ff7675e41ab997ca66cd7af7fd4" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb869d8a50c4226ae0ccead02b414d5" is_pre_election: true
I20260808 04:53:56.678393 8757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b04ff7675e41ab997ca66cd7af7fd4" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39598989aa42659b91e203637b1332" is_pre_election: true
I20260808 04:53:56.678658 8757 raft_consensus.cc:2454] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 0.
I20260808 04:53:56.678673 8543 raft_consensus.cc:2454] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 0.
I20260808 04:53:56.679137 8213 leader_election.cc:302] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc, 0a39598989aa42659b91e203637b1332; no voters:
I20260808 04:53:56.679411 8410 raft_consensus.cc:2781] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:53:56.679492 8410 raft_consensus.cc:482] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:53:56.679538 8410 raft_consensus.cc:3037] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:56.680889 8410 raft_consensus.cc:504] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.681221 8410 leader_election.cc:288] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 election: Requested vote from peers 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317), bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
I20260808 04:53:56.681571 8757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b04ff7675e41ab997ca66cd7af7fd4" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a39598989aa42659b91e203637b1332"
I20260808 04:53:56.681694 8757 raft_consensus.cc:3037] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:56.681792 8543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b04ff7675e41ab997ca66cd7af7fd4" candidate_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb869d8a50c4226ae0ccead02b414d5"
I20260808 04:53:56.681941 8543 raft_consensus.cc:3037] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:53:56.682796 8757 raft_consensus.cc:2454] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 1.
I20260808 04:53:56.682999 8543 raft_consensus.cc:2454] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08399aa6dda34b4ba17a77cbbb51bffc in term 1.
I20260808 04:53:56.683089 8213 leader_election.cc:302] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc, 0a39598989aa42659b91e203637b1332; no voters:
I20260808 04:53:56.683286 8410 raft_consensus.cc:2781] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:53:56.683545 8410 raft_consensus.cc:686] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 LEADER]: Becoming Leader. State: Replica: 08399aa6dda34b4ba17a77cbbb51bffc, State: Running, Role: LEADER
I20260808 04:53:56.683808 8410 consensus_queue.cc:237] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:53:56.686494 8092 catalog_manager.cc:5495] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc reported cstate change: term changed from 0 to 1, leader changed from <none> to 08399aa6dda34b4ba17a77cbbb51bffc (127.15.30.65). New cstate: current_term: 1 leader_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260808 04:53:56.702617 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.699440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.15.30.65:43533)
0808 04:53:56.699549 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 04:53:56.699553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:53:56.699581 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:53:56.699705 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:53:56.699708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:53:56.699759 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:53:56.699845 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:53:56.700042 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.700603 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.701364 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:53:56.701583 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:53:56.701692 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:53:56.701889 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:53:56.702271 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:53:56.702358 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:53:56.702474 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 04:53:56.702519 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 04:53:56.706182 8757 raft_consensus.cc:1261] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Refusing update from remote peer 08399aa6dda34b4ba17a77cbbb51bffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:56.706231 8543 raft_consensus.cc:1261] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Refusing update from remote peer 08399aa6dda34b4ba17a77cbbb51bffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:53:56.706913 8855 consensus_queue.cc:1035] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:56.707111 8410 consensus_queue.cc:1035] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:53:56.718001 8861 mvcc.cc:205] Tried to move back new op lower bound from 7316131171142778880 to 7316131171059281920. Current Snapshot: MvccSnapshot[applied={T|T < 7316131171142778880}]
I20260808 04:53:56.723165 8091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:56.723336 8091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:53:56.725759 8091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Sending DeleteTablet for 3 replicas of tablet f5b04ff7675e41ab997ca66cd7af7fd4
I20260808 04:53:56.726406 8725 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b04ff7675e41ab997ca66cd7af7fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:56 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:38992
I20260808 04:53:56.726464 8294 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b04ff7675e41ab997ca66cd7af7fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:56 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42388
I20260808 04:53:56.726538 8517 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b04ff7675e41ab997ca66cd7af7fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:53:56 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:49446
I20260808 04:53:56.726917 8874 tablet_replica.cc:330] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: stopping tablet replica
I20260808 04:53:56.727129 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 8147
I20260808 04:53:56.727240 8874 raft_consensus.cc:2229] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:56.727540 8874 raft_consensus.cc:2258] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:56.734885 8874 ts_tablet_manager.cc:1905] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:56.739174 8874 ts_tablet_manager.cc:1918] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:56.739327 8874 log.cc:1196] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/f5b04ff7675e41ab997ca66cd7af7fd4
I20260808 04:53:56.739836 8874 ts_tablet_manager.cc:1939] T f5b04ff7675e41ab997ca66cd7af7fd4 P 0a39598989aa42659b91e203637b1332: Deleting consensus metadata
I20260808 04:53:56.741704 8063 catalog_manager.cc:4841] TS 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317): tablet f5b04ff7675e41ab997ca66cd7af7fd4 (table test-table [id=567ebf50d546456c8945486eb597e112]) successfully deleted
I20260808 04:53:56.749353 8875 tablet_replica.cc:330] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: stopping tablet replica
I20260808 04:53:56.749734 8875 raft_consensus.cc:2229] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:53:56.750041 8875 raft_consensus.cc:2258] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:53:56.752755 8875 ts_tablet_manager.cc:1905] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:53:56.754869 8875 ts_tablet_manager.cc:1918] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:53:56.754993 8875 log.cc:1196] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/f5b04ff7675e41ab997ca66cd7af7fd4
I20260808 04:53:56.755342 8875 ts_tablet_manager.cc:1939] T f5b04ff7675e41ab997ca66cd7af7fd4 P bfb869d8a50c4226ae0ccead02b414d5: Deleting consensus metadata
I20260808 04:53:56.756491 8063 catalog_manager.cc:4841] TS bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089): tablet f5b04ff7675e41ab997ca66cd7af7fd4 (table test-table [id=567ebf50d546456c8945486eb597e112]) successfully deleted
W20260808 04:53:56.757771 8067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:43533 (error 108)
I20260808 04:53:56.758241 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 8409
I20260808 04:53:56.760550 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:53:56.760034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:43533 (local address 127.0.0.1:42406)
0808 04:53:56.760296 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 04:53:56.760432 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260808 04:53:56.760772 8067 catalog_manager.cc:4584] TS 08399aa6dda34b4ba17a77cbbb51bffc (127.15.30.65:43533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5b04ff7675e41ab997ca66cd7af7fd4: Network error: Client connection negotiation failed: client connection to 127.15.30.65:43533: connect: Connection refused (error 111)
I20260808 04:53:56.770478 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 8622
I20260808 04:53:56.781418 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 8034
I20260808 04:54:16.791384 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44831
--webserver_interface=127.15.30.126
--webserver_port=44835
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:16.904824 8878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:16.905103 8878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:16.905161 8878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:16.909037 8878 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:54:16.909111 8878 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:54:16.909139 8878 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:54:16.909161 8878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:16.909204 8878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:16.909245 8878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:16.909281 8878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:16.909317 8878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:16.914322 8878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:16.915294 8878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:16.916255 8878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:16.925184 8890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:16.925201 8889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:16.925261 8878 server_base.cc:1034] running on GCE node
W20260808 04:54:16.925207 8892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:16.925992 8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:16.927021 8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:16.928221 8878 hybrid_clock.cc:648] HybridClock initialized: now 1786164856928213 us; error 29 us; skew 500 ppm
Aug 08 04:54:16 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:16.930995 8878 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:54:16.931845 8878 webserver.cc:460] Webserver started at http://127.15.30.126:44835/ using document root <none> and password file <none>
I20260808 04:54:16.932456 8878 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:16.932546 8878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:16.936141 8878 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:16.938339 8905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:16.939582 8878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:16.939728 8878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "846de0155ad1466184e63b32f1268d18"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "35c0436360f4d849d82cee04218c000d"
server_key_iv: "7624b4e5efe178cf3ea05511461c70d3"
server_key_version: "encryptionkey@0"
I20260808 04:54:16.940258 8878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:16.961782 8878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:16.965149 8878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:16.965301 8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:16.972407 8878 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:44831
I20260808 04:54:16.972700 8980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:44831 every 8 connection(s)
I20260808 04:54:16.973506 8878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:16.977728 8981 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:54:16.979184 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 8878
I20260808 04:54:16.979341 8981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Bootstrap starting.
I20260808 04:54:16.979677 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:43533
--local_ip_for_outbound_sockets=127.15.30.65
--tserver_master_addrs=127.15.30.126:44831
--webserver_port=37769
--webserver_interface=127.15.30.65
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:54:17.004055 8981 log.cc:824] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:17.018950 8981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:54:17.019486 8981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Bootstrap complete.
I20260808 04:54:17.025152 8981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:54:17.025669 8981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846de0155ad1466184e63b32f1268d18, State: Initialized, Role: FOLLOWER
I20260808 04:54:17.026232 8981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:54:17.026386 8981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:17.026461 8981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:17.026557 8981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:54:17.029088 8981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:54:17.029443 8981 leader_election.cc:302] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846de0155ad1466184e63b32f1268d18; no voters:
I20260808 04:54:17.029788 8981 leader_election.cc:288] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:54:17.029917 8990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:54:17.030197 8990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [term 3 LEADER]: Becoming Leader. State: Replica: 846de0155ad1466184e63b32f1268d18, State: Running, Role: LEADER
I20260808 04:54:17.030519 8990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } }
I20260808 04:54:17.031080 8981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:17.032157 8992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "846de0155ad1466184e63b32f1268d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } } }
I20260808 04:54:17.032157 8993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846de0155ad1466184e63b32f1268d18. Latest consensus state: current_term: 3 leader_uuid: "846de0155ad1466184e63b32f1268d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846de0155ad1466184e63b32f1268d18" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 44831 } } }
I20260808 04:54:17.032315 8993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:17.032315 8992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:17.033797 9006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:17.038671 9006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:17.039343 9006 catalog_manager.cc:1259] Loaded cluster ID: c9f6e7f77f584daea01459f088ff4985
I20260808 04:54:17.039407 9006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:17.041986 9006 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:17.042748 9006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Loaded TSK: 0
I20260808 04:54:17.042837 9006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 846de0155ad1466184e63b32f1268d18: Loaded TSK: 1
F20260808 04:54:17.043747 9006 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd7f23399ad google::LogMessage::Fail() at ??:0
@ 0x7fd7f233c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd7f23394de google::LogMessage::Flush() at ??:0
@ 0x7fd7f233aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd7f4ffd18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd7f4ff5f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd7fcc9e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd7fcc7899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd7fcc79178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd7fccbadf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd7fccd982b std::function<>::operator()() at ??:0
@ 0x7fd7fcc78b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd7fcc79f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd7fcc76579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd7fccba640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd7f458b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd7f458bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd7f458d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd7f457c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd7f5bc06db start_thread at ??:0
@ 0x7fd7f0e3761f clone at ??:0
W20260808 04:54:17.133961 8987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:17.134176 8987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:17.134217 8987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:17.138423 8987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:17.138509 8987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:17.138619 8987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:54:17.143913 8987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:43533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=37769
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:17.145071 8987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:17.146114 8987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:17.155781 9066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:17.155782 9060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:17.156052 8987 server_base.cc:1034] running on GCE node
W20260808 04:54:17.156241 9061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:17.156726 8987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:17.157469 8987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:17.158670 8987 hybrid_clock.cc:648] HybridClock initialized: now 1786164857158651 us; error 35 us; skew 500 ppm
Aug 08 04:54:17 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:17.161581 8987 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:54:17.162490 8987 webserver.cc:460] Webserver started at http://127.15.30.65:37769/ using document root <none> and password file <none>
I20260808 04:54:17.163057 8987 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:17.163120 8987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:17.167237 8987 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:54:17.169664 9076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:17.171025 8987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:54:17.171193 8987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "08399aa6dda34b4ba17a77cbbb51bffc"
format_stamp: "Formatted at 2026-08-08 04:53:34 on dist-test-slave-zpvx"
server_key: "a223d8a89fe3c2cd16260ab0340eadd8"
server_key_iv: "c2eb83af2494bfdd6283a3896fe36a97"
server_key_version: "encryptionkey@0"
I20260808 04:54:17.171756 8987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:17.185922 8987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:17.189124 8987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:17.189347 8987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:17.190130 8987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:17.191314 9086 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:54:17.196863 8987 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:54:17.196961 8987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 04:54:17.197065 8987 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:54:17.198706 8987 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:54:17.198752 8987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:54:17.199086 9086 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap starting.
I20260808 04:54:17.209540 8987 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:43533
I20260808 04:54:17.210443 8987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:54:17.211489 9245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:43533 every 8 connection(s)
I20260808 04:54:17.218396 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 8987
I20260808 04:54:17.218879 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:41089
--local_ip_for_outbound_sockets=127.15.30.66
--tserver_master_addrs=127.15.30.126:44831
--webserver_port=34177
--webserver_interface=127.15.30.66
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:54:17.221613 9086 log.cc:824] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:17.228466 9086 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:54:17.228843 9086 tablet_bootstrap.cc:492] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Bootstrap complete.
I20260808 04:54:17.229997 9086 ts_tablet_manager.cc:1397] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20260808 04:54:17.232254 9086 raft_consensus.cc:348] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:54:17.232795 9086 raft_consensus.cc:729] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08399aa6dda34b4ba17a77cbbb51bffc, State: Initialized, Role: FOLLOWER
I20260808 04:54:17.233291 9086 consensus_queue.cc:260] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:54:17.234114 9086 ts_tablet_manager.cc:1428] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260808 04:54:17.318074 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:54:17.212568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:35565)
0808 04:54:17.213028 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.213080 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:17.213879 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:17.317137 (+103258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":142,"threads_started":1}
W20260808 04:54:17.318706 9247 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:44831 (0 consecutive failures): Network error: Failed to ping master at 127.15.30.126:44831: Client connection negotiation failed: client connection to 127.15.30.126:44831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:54:17.319478 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.319209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:59797)
0808 04:54:17.319351 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.319426 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 04:54:17.320232 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.320028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:49271)
0808 04:54:17.320137 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.320188 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:54:17.320493 9247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:54:17.364018 9252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:17.364274 9252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:17.364347 9252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:17.368249 9252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:17.368328 9252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:17.368440 9252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:54:17.373473 9252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:41089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=34177
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:17.374454 9252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:17.375422 9252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:17.384821 9264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:17.384864 9265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:17.384835 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:17.384934 9252 server_base.cc:1034] running on GCE node
I20260808 04:54:17.385440 9252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:17.386024 9252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:17.387183 9252 hybrid_clock.cc:648] HybridClock initialized: now 1786164857387166 us; error 33 us; skew 500 ppm
Aug 08 04:54:17 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:17.392935 9252 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:54:17.393759 9252 webserver.cc:460] Webserver started at http://127.15.30.66:34177/ using document root <none> and password file <none>
I20260808 04:54:17.394302 9252 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:17.394366 9252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:17.398028 9252 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:17.400012 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:17.401258 9252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:17.401391 9252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "bfb869d8a50c4226ae0ccead02b414d5"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "c7b0c2f64bc66e362e837e9f58869dda"
server_key_iv: "ba6e956b28408ff4c4b5ccdfa6ec95e5"
server_key_version: "encryptionkey@0"
I20260808 04:54:17.401911 9252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:17.415695 9252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:17.418885 9252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:17.419078 9252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:17.419628 9252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:17.420468 9252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:17.420534 9252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:17.420603 9252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:17.420645 9252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:17.429497 9252 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:41089
I20260808 04:54:17.429529 9455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:41089 every 8 connection(s)
I20260808 04:54:17.430424 9252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:54:17.431293 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 9252
I20260808 04:54:17.431733 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:41317
--local_ip_for_outbound_sockets=127.15.30.67
--tserver_master_addrs=127.15.30.126:44831
--webserver_port=38499
--webserver_interface=127.15.30.67
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:54:17.434415 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.432603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:45621)
0808 04:54:17.433081 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.433208 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
W20260808 04:54:17.434854 9456 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:44831 (0 consecutive failures): Network error: Failed to ping master at 127.15.30.126:44831: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
I20260808 04:54:17.435467 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.435255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:49613)
0808 04:54:17.435365 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.435412 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 04:54:17.436141 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.435958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:51365)
0808 04:54:17.436055 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.436095 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:54:17.436349 9456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:54:17.544989 9460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:17.545212 9460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:17.545290 9460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:17.549759 9460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:17.549809 9460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:17.549909 9460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:54:17.554955 9460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:41317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=38499
--tserver_master_addrs=127.15.30.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:17.555939 9460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:17.556960 9460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:17.566488 9470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:17.566504 9466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:17.566500 9467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:17.566787 9460 server_base.cc:1034] running on GCE node
I20260808 04:54:17.567226 9460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:17.567855 9460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:17.569046 9460 hybrid_clock.cc:648] HybridClock initialized: now 1786164857569019 us; error 35 us; skew 500 ppm
Aug 08 04:54:17 dist-test-slave-zpvx krb5kdc[7143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:17.571673 9460 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:54:17.572584 9460 webserver.cc:460] Webserver started at http://127.15.30.67:38499/ using document root <none> and password file <none>
I20260808 04:54:17.573133 9460 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:17.573226 9460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:17.577075 9460 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:54:17.579108 9480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:17.580250 9460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:17.580421 9460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "0a39598989aa42659b91e203637b1332"
format_stamp: "Formatted at 2026-08-08 04:53:35 on dist-test-slave-zpvx"
server_key: "a97e15bf66382ec23dcf3058dd84b377"
server_key_iv: "1b5daf5ce7b1e748ce2b864ab5eeea02"
server_key_version: "encryptionkey@0"
I20260808 04:54:17.580984 9460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:17.602267 9460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:17.605461 9460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:17.605661 9460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:17.606271 9460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:17.607039 9460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:17.607086 9460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:17.607124 9460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:17.607146 9460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:17.616698 9460 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:41317
I20260808 04:54:17.616731 9669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:41317 every 8 connection(s)
I20260808 04:54:17.617691 9460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:54:17.619879 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 9460
I20260808 04:54:17.621937 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.619890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:58313)
0808 04:54:17.620531 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.620682 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
W20260808 04:54:17.622393 9670 heartbeater.cc:644] Failed to heartbeat to 127.15.30.126:44831 (0 consecutive failures): Network error: Failed to ping master at 127.15.30.126:44831: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
I20260808 04:54:17.622968 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.622772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:59121)
0808 04:54:17.622870 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.622917 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 04:54:17.623598 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:17.623422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:38843)
0808 04:54:17.623512 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:17.623552 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 04:54:17.623845 9670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:54:18.321435 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:18.321004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:40493)
0808 04:54:18.321261 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:18.321320 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260808 04:54:18.437290 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:18.436891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:36599)
0808 04:54:18.437127 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:18.437178 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
I20260808 04:54:18.624835 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:18.624450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:55129)
0808 04:54:18.624679 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:18.624733 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260808 04:54:19.012499 9683 raft_consensus.cc:482] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:19.012790 9683 raft_consensus.cc:504] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:54:19.013883 9683 leader_election.cc:288] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317), bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
Aug 08 04:54:19 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.67@KRBTEST.COM
Aug 08 04:54:19 dist-test-slave-zpvx krb5kdc[7143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.66@KRBTEST.COM
I20260808 04:54:19.027944 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:19.014105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:41317 (local address 127.15.30.65:59731)
0808 04:54:19.014352 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:19.014369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:19.014477 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:19.016341 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:19.016347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:19.016941 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:19.017435 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:19.017452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.018740 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.018748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:19.019658 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:19.019669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.020401 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.020405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:19.020628 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:19.021314 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:19.021325 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:19.023206 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:19.025540 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:19.025545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:19.025558 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:19.025874 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:19.026227 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:19.026231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:19.026233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:19.026371 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:19.026964 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:19.026970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:19.027208 (+ 238us) client_negotiation.cc:769] Sending connection context
0808 04:54:19.027390 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 04:54:19.027724 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":167,"spinlock_wait_cycles":7552,"thread_start_us":63,"threads_started":1}
I20260808 04:54:19.027953 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:19.014307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:59731 (local address 127.15.30.67:41317)
0808 04:54:19.014784 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:19.014819 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:19.014875 (+ 56us) server_negotiation.cc:407] Connection header received
0808 04:54:19.015605 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:19.015610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:19.015954 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:19.016265 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:19.017619 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.018585 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.019797 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.020250 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.020455 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:19.023471 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:19.023476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:19.023487 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:19.023514 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:19.025397 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:19.025995 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:19.026003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:19.026012 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:19.026127 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:19.026468 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:19.026473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:19.026474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:19.026845 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:19.026989 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:19.027502 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 04:54:19.027763 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":55,"threads_started":1}
I20260808 04:54:19.027979 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:19.014639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:58559 (local address 127.15.30.66:41089)
0808 04:54:19.014962 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:19.014973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:19.015007 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:54:19.015636 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:19.015641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:19.015949 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:19.016198 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:19.017621 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.018587 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.019815 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.020266 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.020455 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:19.023826 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:19.023831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:19.023842 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:19.023870 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:19.025512 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:19.026030 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:19.026036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:19.026048 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:19.026159 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:19.026476 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:19.026480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:19.026482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:19.026845 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:19.026992 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:19.027509 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 04:54:19.027790 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":132,"threads_started":1}
I20260808 04:54:19.027981 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:19.014464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:41089 (local address 127.15.30.65:58559)
0808 04:54:19.014790 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:19.014806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:19.014911 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:19.016489 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:19.016492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:19.016942 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:19.017435 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:19.017451 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.018738 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.018750 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:19.019656 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:19.019665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:19.020415 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:19.020419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:19.020632 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:19.021270 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:19.021297 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:19.023630 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:19.025635 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:19.025640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:19.025642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:19.025909 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:19.026265 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:19.026269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:19.026271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:19.026377 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:19.026967 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:19.026970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:19.027213 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 04:54:19.027390 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 04:54:19.027821 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":327,"spinlock_wait_cycles":45056,"thread_start_us":84,"threads_started":1}
W20260808 04:54:19.030368 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
W20260808 04:54:19.030597 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
I20260808 04:54:19.030654 9077 leader_election.cc:302] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc; no voters: 0a39598989aa42659b91e203637b1332, bfb869d8a50c4226ae0ccead02b414d5
I20260808 04:54:19.030954 9683 raft_consensus.cc:2726] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:54:19.322429 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:19.322135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:59109)
0808 04:54:19.322279 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:19.322344 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 04:54:19.438449 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:19.437990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:34773)
0808 04:54:19.438270 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:19.438338 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260808 04:54:19.625864 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:19.625489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:48361)
0808 04:54:19.625724 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:19.625778 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260808 04:54:20.323634 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:20.323147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:52581)
0808 04:54:20.323394 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:20.323495 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260808 04:54:20.439653 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:20.439237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:55873)
0808 04:54:20.439505 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:20.439560 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260808 04:54:20.626966 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:20.626568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:46835)
0808 04:54:20.626816 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:20.626871 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260808 04:54:21.020711 9700 raft_consensus.cc:482] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:21.020877 9700 raft_consensus.cc:504] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:54:21.021289 9700 leader_election.cc:288] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317), bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
W20260808 04:54:21.022248 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
W20260808 04:54:21.022398 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
I20260808 04:54:21.022528 9077 leader_election.cc:302] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc; no voters: 0a39598989aa42659b91e203637b1332, bfb869d8a50c4226ae0ccead02b414d5
I20260808 04:54:21.022712 9700 raft_consensus.cc:2726] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:54:21.324857 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:21.324444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:33053)
0808 04:54:21.324698 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:21.324756 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260808 04:54:21.440819 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:21.440333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:38409)
0808 04:54:21.440647 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:21.440699 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":143,"threads_started":1}
I20260808 04:54:21.628129 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:21.627703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:50437)
0808 04:54:21.627987 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:21.628038 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":114,"threads_started":1}
I20260808 04:54:22.326479 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:22.325856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:37645)
0808 04:54:22.326265 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:22.326340 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":100,"threads_started":1}
I20260808 04:54:22.441934 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:22.441569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:53549)
0808 04:54:22.441797 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:22.441849 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260808 04:54:22.629405 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:22.628928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:41567)
0808 04:54:22.629194 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:22.629272 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260808 04:54:23.327729 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:23.327322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:53155)
0808 04:54:23.327581 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:23.327634 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260808 04:54:23.443234 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:23.442740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:48201)
0808 04:54:23.443078 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:23.443137 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":74,"threads_started":1}
I20260808 04:54:23.630658 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:23.630208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:43651)
0808 04:54:23.630511 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:23.630565 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":85,"threads_started":1}
I20260808 04:54:23.887692 9713 raft_consensus.cc:482] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:23.887840 9713 raft_consensus.cc:504] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:54:23.888300 9713 leader_election.cc:288] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317), bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
W20260808 04:54:23.889147 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
W20260808 04:54:23.889361 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
I20260808 04:54:23.889439 9077 leader_election.cc:302] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc; no voters: 0a39598989aa42659b91e203637b1332, bfb869d8a50c4226ae0ccead02b414d5
I20260808 04:54:23.889647 9713 raft_consensus.cc:2726] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:54:24.328917 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:24.328527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:42109)
0808 04:54:24.328769 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:24.328820 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260808 04:54:24.444561 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:24.443986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:41625)
0808 04:54:24.444358 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:24.444472 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":130,"threads_started":1}
I20260808 04:54:24.631806 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:24.631362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:44313)
0808 04:54:24.631652 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:24.631708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":74,"threads_started":1}
I20260808 04:54:25.330056 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:25.329642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:58667)
0808 04:54:25.329906 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:25.329960 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260808 04:54:25.445642 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:25.445252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:44587)
0808 04:54:25.445495 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:25.445550 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260808 04:54:25.632999 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:25.632568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:39479)
0808 04:54:25.632851 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:25.632906 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
I20260808 04:54:26.331182 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:26.330766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:44041)
0808 04:54:26.331035 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:26.331086 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260808 04:54:26.446877 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:26.446428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:38191)
0808 04:54:26.446713 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:26.446776 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260808 04:54:26.634115 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:26.633707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:50311)
0808 04:54:26.633966 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:26.634020 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260808 04:54:27.140103 9728 raft_consensus.cc:482] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:27.140268 9728 raft_consensus.cc:504] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a39598989aa42659b91e203637b1332" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 41317 } } peers { permanent_uuid: "bfb869d8a50c4226ae0ccead02b414d5" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 41089 } } peers { permanent_uuid: "08399aa6dda34b4ba17a77cbbb51bffc" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 43533 } }
I20260808 04:54:27.140709 9728 leader_election.cc:288] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317), bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089)
W20260808 04:54:27.141440 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a39598989aa42659b91e203637b1332 (127.15.30.67:41317): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
W20260808 04:54:27.141561 9077 leader_election.cc:341] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfb869d8a50c4226ae0ccead02b414d5 (127.15.30.66:41089): Not found: Tablet not found: f5b04ff7675e41ab997ca66cd7af7fd4
I20260808 04:54:27.141602 9077 leader_election.cc:302] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08399aa6dda34b4ba17a77cbbb51bffc; no voters: 0a39598989aa42659b91e203637b1332, bfb869d8a50c4226ae0ccead02b414d5
I20260808 04:54:27.141734 9728 raft_consensus.cc:2726] T f5b04ff7675e41ab997ca66cd7af7fd4 P 08399aa6dda34b4ba17a77cbbb51bffc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:54:27.332523 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:27.332042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:41525)
0808 04:54:27.332303 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:27.332360 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260808 04:54:27.448033 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:27.447620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:50557)
0808 04:54:27.447889 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:27.447941 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260808 04:54:27.635424 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:27.634945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:35725)
0808 04:54:27.635279 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:27.635332 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":74,"threads_started":1}
I20260808 04:54:28.333809 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:28.333390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:37619)
0808 04:54:28.333652 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:28.333709 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260808 04:54:28.449190 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:28.448796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:48225)
0808 04:54:28.449044 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:28.449099 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260808 04:54:28.636626 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:28.636233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:45309)
0808 04:54:28.636480 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:28.636533 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260808 04:54:29.335037 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:29.334546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:47329)
0808 04:54:29.334859 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:29.334922 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260808 04:54:29.450551 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:29.450106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:43811)
0808 04:54:29.450400 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:29.450456 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260808 04:54:29.637936 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:29.637423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:59035)
0808 04:54:29.637725 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:29.637818 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":114,"threads_started":1}
I20260808 04:54:30.336279 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:30.335815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:49463)
0808 04:54:30.336118 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:30.336178 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260808 04:54:30.451656 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:30.451262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:49943)
0808 04:54:30.451503 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:30.451560 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260808 04:54:30.639077 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:30.638688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:34645)
0808 04:54:30.638930 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:30.638981 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260808 04:54:31.337499 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:31.337095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:51185)
0808 04:54:31.337347 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:31.337402 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260808 04:54:31.452986 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:31.452582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:49561)
0808 04:54:31.452827 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:31.452884 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260808 04:54:31.640322 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:31.639883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:51077)
0808 04:54:31.640159 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:31.640215 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260808 04:54:32.339112 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:32.338574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.65:46767)
0808 04:54:32.338928 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:32.338994 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260808 04:54:32.454320 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:32.453797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.66:57917)
0808 04:54:32.454143 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:32.454200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":70,"threads_started":1}
I20260808 04:54:32.623051 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 8987
I20260808 04:54:32.633832 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 9252
I20260808 04:54:32.643712 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:32.642345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:44831 (local address 127.15.30.67:43513)
0808 04:54:32.643514 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:32.643583 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":812,"threads_started":1}
I20260808 04:54:32.649622 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 9460
2026-08-08T04:54:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59473 ms)
[----------] 30 tests from SecurityITest (252108 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:32 dist-test-slave-zpvx krb5kdc[9766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:32 dist-test-slave-zpvx krb5kdc[9766](info): set up 2 sockets
Aug 08 04:54:32 dist-test-slave-zpvx krb5kdc[9766](info): commencing operation
krb5kdc: starting...
W20260808 04:54:33.911109 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:33 dist-test-slave-zpvx krb5kdc[9766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:54:34.033365 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34193
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:34193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:34.145201 9781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:34.145437 9781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:34.145474 9781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:34.149288 9781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:34.149333 9781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:34.149351 9781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:34.149367 9781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:34.149382 9781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:34.154459 9781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:34193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:34.155347 9781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:34.156256 9781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:34.164722 9789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:34.164734 9791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:34.164856 9788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:34.164808 9781 server_base.cc:1034] running on GCE node
I20260808 04:54:34.165534 9781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:34.166553 9781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:34.167747 9781 hybrid_clock.cc:648] HybridClock initialized: now 1786164874167714 us; error 46 us; skew 500 ppm
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:34.170455 9781 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:54:34.171277 9781 webserver.cc:460] Webserver started at http://127.15.30.126:32861/ using document root <none> and password file <none>
I20260808 04:54:34.171770 9781 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:34.171829 9781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:34.171983 9781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:34.174466 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "03070f429c3f4f77a155310e9f4245a0"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "31aa03d8fd95f4aa157007c3871209ef"
server_key_iv: "abd7a515793d8558fba7b3971ce39dd3"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.174959 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "03070f429c3f4f77a155310e9f4245a0"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "31aa03d8fd95f4aa157007c3871209ef"
server_key_iv: "abd7a515793d8558fba7b3971ce39dd3"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.178522 9781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:34.180776 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.181970 9781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:34.182152 9781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "03070f429c3f4f77a155310e9f4245a0"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "31aa03d8fd95f4aa157007c3871209ef"
server_key_iv: "abd7a515793d8558fba7b3971ce39dd3"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.182267 9781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:34.203613 9781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:34.206485 9781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:34.206693 9781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:34.214010 9781 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:34193
I20260808 04:54:34.214032 9888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:34193 every 8 connection(s)
I20260808 04:54:34.214908 9781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:34.217581 9889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:34.219121 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 9781
I20260808 04:54:34.219259 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:34.219547 15481 external_mini_cluster.cc:1428] Setting key 1b8029f2d7bfde803f5a2de9ad3823c5
I20260808 04:54:34.223520 9889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0: Bootstrap starting.
I20260808 04:54:34.225714 9889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:34.226472 9889 log.cc:824] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:34.228580 9889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0: No bootstrap required, opened a new log
I20260808 04:54:34.232733 9889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03070f429c3f4f77a155310e9f4245a0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34193 } }
I20260808 04:54:34.232936 9889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:34.233014 9889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03070f429c3f4f77a155310e9f4245a0, State: Initialized, Role: FOLLOWER
I20260808 04:54:34.233433 9889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03070f429c3f4f77a155310e9f4245a0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34193 } }
I20260808 04:54:34.233547 9889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:34.233621 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.220944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.15.30.126:34193)
0808 04:54:34.221370 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:34.221384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:34.221418 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:54:34.222045 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:34.222061 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:34.222375 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:34.222699 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:34.223514 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.224627 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.225476 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.225991 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.226168 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.228876 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:34.228881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:34.228893 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:34.228919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:34.230561 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.231077 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.231081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.231085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.231213 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.231499 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.231501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.231503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.231847 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:34.232004 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:34.232304 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 04:54:34.232670 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260808 04:54:34.233629 9889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:34.233800 9889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:34.234647 9889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03070f429c3f4f77a155310e9f4245a0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34193 } }
I20260808 04:54:34.234987 9889 leader_election.cc:302] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03070f429c3f4f77a155310e9f4245a0; no voters:
I20260808 04:54:34.235318 9889 leader_election.cc:288] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:34.235430 9897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:34.235672 9897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [term 1 LEADER]: Becoming Leader. State: Replica: 03070f429c3f4f77a155310e9f4245a0, State: Running, Role: LEADER
I20260808 04:54:34.235980 9897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03070f429c3f4f77a155310e9f4245a0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34193 } }
I20260808 04:54:34.236531 9889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:34.237185 9900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03070f429c3f4f77a155310e9f4245a0. Latest consensus state: current_term: 1 leader_uuid: "03070f429c3f4f77a155310e9f4245a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03070f429c3f4f77a155310e9f4245a0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34193 } } }
I20260808 04:54:34.237314 9900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:34.237711 9906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:34.237675 9898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03070f429c3f4f77a155310e9f4245a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03070f429c3f4f77a155310e9f4245a0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34193 } } }
I20260808 04:54:34.237788 9898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:34.240516 9906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:34.246002 9906 catalog_manager.cc:1347] Generated new cluster ID: d66bb481fa4c452ca258244dc1a8b3c2
I20260808 04:54:34.246078 9906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:34.257865 9906 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:34.258716 9906 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:34.265489 9906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0: Generated new TSK 0
I20260808 04:54:34.266140 9906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:54:34.303833 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34193
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:34.419906 9926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:34.420147 9926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:34.420187 9926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:34.424122 9926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:34.424172 9926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:34.424274 9926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:54:34.429332 9926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:34.430343 9926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:34.431335 9926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:34.441036 9938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:34.441061 9936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:34.441036 9940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:34.441282 9926 server_base.cc:1034] running on GCE node
I20260808 04:54:34.441813 9926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:34.442430 9926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:34.443610 9926 hybrid_clock.cc:648] HybridClock initialized: now 1786164874443596 us; error 26 us; skew 500 ppm
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:34.446465 9926 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:54:34.447292 9926 webserver.cc:460] Webserver started at http://127.15.30.65:34643/ using document root <none> and password file <none>
I20260808 04:54:34.447852 9926 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:34.447947 9926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:34.448171 9926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:34.450553 9926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "c043253988ac4e7f975b7f50f9863a76"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "ce4882f53fe72f1d6a2245639eb697a0"
server_key_iv: "ad8ca1532abaecaf95e6e4f830ba452d"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.451215 9926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "c043253988ac4e7f975b7f50f9863a76"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "ce4882f53fe72f1d6a2245639eb697a0"
server_key_iv: "ad8ca1532abaecaf95e6e4f830ba452d"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.455296 9926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 04:54:34.457890 9951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.459270 9926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:54:34.459404 9926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "c043253988ac4e7f975b7f50f9863a76"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "ce4882f53fe72f1d6a2245639eb697a0"
server_key_iv: "ad8ca1532abaecaf95e6e4f830ba452d"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.459519 9926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:34.474416 9926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:34.477909 9926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:34.478128 9926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:34.478762 9926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:34.479642 9926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:34.479696 9926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.479769 9926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:34.479804 9926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.489606 9926 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:40581
I20260808 04:54:34.489624 10134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:40581 every 8 connection(s)
I20260808 04:54:34.490562 9926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:54:34.490869 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 9926
I20260808 04:54:34.491034 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:54:34.491412 15481 external_mini_cluster.cc:1428] Setting key e462a8df15cd053740086f49b49cbd8a
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:54:34.506125 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.492556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:43949 (local address 127.15.30.126:34193)
0808 04:54:34.492708 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:34.492712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:34.494070 (+ 1358us) server_negotiation.cc:407] Connection header received
0808 04:54:34.495448 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:34.495454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:34.495523 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:34.495631 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:34.496847 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.497807 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.499039 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.499521 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.499671 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.502152 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:34.502158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:34.502162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:34.502193 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:34.503964 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.504520 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.504523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.504525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.504611 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.504942 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.504945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.504946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.505134 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:34.505215 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:34.505835 (+ 620us) server_negotiation.cc:299] Negotiation successful
0808 04:54:34.505963 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:54:34.506914 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.493079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34193 (local address 127.15.30.65:43949)
0808 04:54:34.493895 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:34.493953 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:34.495183 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:34.495784 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:34.495790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:34.496132 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:34.496692 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:34.496707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.497939 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.497942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:34.498903 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:34.498913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.499683 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.499686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:34.499818 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.500508 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:34.500524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:34.501958 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:34.504082 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:34.504087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:34.504098 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:34.504346 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:34.504707 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:34.504710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:34.504711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:34.504844 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:34.505263 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:34.505268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:34.505484 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 04:54:34.505694 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 04:54:34.505954 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":123,"threads_started":1}
I20260808 04:54:34.508159 10136 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34193
I20260808 04:54:34.508476 10136 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:34.509006 10136 heartbeater.cc:511] Master 127.15.30.126:34193 requested a full tablet report, sending...
I20260808 04:54:34.510705 9829 ts_manager.cc:194] Registered new tserver with Master: c043253988ac4e7f975b7f50f9863a76 (127.15.30.65:40581)
I20260808 04:54:34.511891 9829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:43949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:54:34.523582 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34193
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:34.635644 10143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:34.635878 10143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:34.635918 10143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:34.640304 10143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:34.640359 10143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:34.640506 10143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:54:34.645533 10143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:34.646471 10143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:34.647442 10143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:34.656782 10155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:34.656853 10143 server_base.cc:1034] running on GCE node
W20260808 04:54:34.656941 10151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:34.656976 10150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:34.657366 10143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:34.657991 10143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:34.659193 10143 hybrid_clock.cc:648] HybridClock initialized: now 1786164874659157 us; error 43 us; skew 500 ppm
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:34.662009 10143 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:54:34.662920 10143 webserver.cc:460] Webserver started at http://127.15.30.66:32887/ using document root <none> and password file <none>
I20260808 04:54:34.663507 10143 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:34.663620 10143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:34.663849 10143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:34.665937 10143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "ec19f6126f054d0894dd0dff464ed85e"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "79d1fe227cc1d9c43a711a1e5d73c30f"
server_key_iv: "6a016e0a0003e756e805b9be65edafe5"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.666519 10143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "ec19f6126f054d0894dd0dff464ed85e"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "79d1fe227cc1d9c43a711a1e5d73c30f"
server_key_iv: "6a016e0a0003e756e805b9be65edafe5"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.670290 10143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:54:34.672664 10166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.673875 10143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:54:34.674008 10143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "ec19f6126f054d0894dd0dff464ed85e"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "79d1fe227cc1d9c43a711a1e5d73c30f"
server_key_iv: "6a016e0a0003e756e805b9be65edafe5"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.674149 10143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:34.706832 10143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:34.710179 10143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:34.710402 10143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:34.711010 10143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:34.711861 10143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:34.711934 10143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.712000 10143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:34.712045 10143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.720767 10143 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:39351
I20260808 04:54:34.720814 10352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:39351 every 8 connection(s)
I20260808 04:54:34.721625 10143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:54:34.730649 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 10143
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:34.730813 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:54:34.731168 15481 external_mini_cluster.cc:1428] Setting key 53fbd40856ebf3ee105b30347759e925
I20260808 04:54:34.735646 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.723470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:46337 (local address 127.15.30.126:34193)
0808 04:54:34.723614 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:34.723618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:34.724673 (+ 1055us) server_negotiation.cc:407] Connection header received
0808 04:54:34.725567 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:34.725571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:34.725625 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:34.725732 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:34.726839 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.727370 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.728553 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.728776 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.728887 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.731546 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:34.731549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:34.731551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:34.731578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:34.733378 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.733896 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.733903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.733907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.734005 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.734368 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.734374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.734379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.734630 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:34.734738 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:34.735415 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 04:54:34.735522 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:54:34.736770 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.723827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34193 (local address 127.15.30.66:46337)
0808 04:54:34.724529 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:34.724577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:34.725371 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:34.725901 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:34.725908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:34.726238 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:34.726696 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:34.726709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.727509 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.727517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:34.728433 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:34.728443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.728928 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.728932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:34.729108 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.729677 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:34.729693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:34.731303 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:34.733513 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:34.733519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:34.733529 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:34.733787 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:34.734134 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:34.734137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:34.734138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:34.734271 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:34.734734 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:34.734766 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:34.735139 (+ 373us) client_negotiation.cc:769] Sending connection context
0808 04:54:34.735369 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 04:54:34.735711 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":114,"threads_started":1}
I20260808 04:54:34.738044 10354 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34193
I20260808 04:54:34.738298 10354 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:34.738848 10354 heartbeater.cc:511] Master 127.15.30.126:34193 requested a full tablet report, sending...
I20260808 04:54:34.739919 9829 ts_manager.cc:194] Registered new tserver with Master: ec19f6126f054d0894dd0dff464ed85e (127.15.30.66:39351)
I20260808 04:54:34.740751 9829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:46337
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:54:34.761001 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34193
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:34.876674 10359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:34.876935 10359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:34.877007 10359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:34.881219 10359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:34.881271 10359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:34.881350 10359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:54:34.886648 10359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:34193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:34.887647 10359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:34.888665 10359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:34.898012 10372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:34.898012 10367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:34.898082 10368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:34.899339 10359 server_base.cc:1034] running on GCE node
I20260808 04:54:34.899731 10359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:34.900328 10359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:34.901633 10359 hybrid_clock.cc:648] HybridClock initialized: now 1786164874901598 us; error 55 us; skew 500 ppm
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:34.904486 10359 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:54:34.905416 10359 webserver.cc:460] Webserver started at http://127.15.30.67:42789/ using document root <none> and password file <none>
I20260808 04:54:34.905968 10359 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:34.906030 10359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:34.906251 10359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:34.908299 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "b9ded059704d4f218985281966ddece5"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "85cfc28777cd34d93647f162d1279206"
server_key_iv: "a8496191e83082bf81fbcf98c65acaa3"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.908921 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "b9ded059704d4f218985281966ddece5"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "85cfc28777cd34d93647f162d1279206"
server_key_iv: "a8496191e83082bf81fbcf98c65acaa3"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.912688 10359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:54:34.915227 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.916621 10359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:54:34.916756 10359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "b9ded059704d4f218985281966ddece5"
format_stamp: "Formatted at 2026-08-08 04:54:34 on dist-test-slave-zpvx"
server_key: "85cfc28777cd34d93647f162d1279206"
server_key_iv: "a8496191e83082bf81fbcf98c65acaa3"
server_key_version: "encryptionkey@0"
I20260808 04:54:34.916867 10359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:34.934904 10359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:34.938243 10359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:34.938480 10359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:34.939038 10359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:34.939885 10359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:34.939957 10359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.940027 10359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:34.940081 10359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:34.951012 10359 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:38445
I20260808 04:54:34.951052 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:38445 every 8 connection(s)
I20260808 04:54:34.951900 10359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:54:34.958549 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 10359
I20260808 04:54:34.958695 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:54:34.959028 15481 external_mini_cluster.cc:1428] Setting key afe5e8ad5de71ef31c6ddb48fb0db82c
Aug 08 04:54:34 dist-test-slave-zpvx krb5kdc[9766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:34.967201 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.954013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:40585 (local address 127.15.30.126:34193)
0808 04:54:34.954147 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:34.954151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:34.955405 (+ 1254us) server_negotiation.cc:407] Connection header received
0808 04:54:34.956297 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:34.956300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:34.956363 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:34.956495 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:34.957729 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.958295 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.959960 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.960185 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.960301 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.963626 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:34.963630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:34.963632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:34.963656 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:34.965165 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.965681 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.965686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.965687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.965780 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.966138 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.966143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.966145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.966330 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:34.966443 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:34.966914 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 04:54:34.967035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:54:34.967938 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.954531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:34193 (local address 127.15.30.67:40585)
0808 04:54:34.955253 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:34.955297 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:34.956085 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:34.956671 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:34.956678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:34.957021 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:34.957576 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:34.957594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.958464 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.958469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:34.959783 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:34.959795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.960358 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.960364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:34.960647 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.961701 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:34.961718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:34.963445 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:34.965296 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:34.965302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:34.965316 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:34.965572 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:34.965878 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:34.965881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:34.965884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:34.966036 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:34.966444 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:34.966452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:34.966683 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 04:54:34.966868 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 04:54:34.967164 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":159,"threads_started":1}
I20260808 04:54:34.969219 10586 heartbeater.cc:347] Connected to a master server at 127.15.30.126:34193
I20260808 04:54:34.969460 10586 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:34.969928 10586 heartbeater.cc:511] Master 127.15.30.126:34193 requested a full tablet report, sending...
I20260808 04:54:34.970986 9829 ts_manager.cc:194] Registered new tserver with Master: b9ded059704d4f218985281966ddece5 (127.15.30.67:38445)
I20260808 04:54:34.971544 9829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:40585
I20260808 04:54:34.973896 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:54:34.984082 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.975897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.15.30.126:34193)
0808 04:54:34.976029 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:34.976036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:34.976269 (+ 233us) server_negotiation.cc:407] Connection header received
0808 04:54:34.976349 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:34.976354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:34.976440 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:34.976535 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:34.977189 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.978158 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.979110 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:34.979557 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:34.979682 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:34.981031 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:34.981035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:34.981037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:34.981062 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:34.982411 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.982839 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.982843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.982844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.982930 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:34.983212 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:34.983215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:34.983217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:34.983430 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:34.983574 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:34.983822 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 04:54:34.983931 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:54:34.987106 9829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:54:34.990216 9829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:54:35.007115 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:34.999583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:40581 (local address 127.0.0.1:52110)
0808 04:54:34.999923 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:34.999962 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:35.000069 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:35.002189 (+ 2120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:35.002195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:35.002230 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:35.002371 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.002380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.003817 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.003821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.005262 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.005279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.006572 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.006575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.006685 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.006722 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:54:35.006829 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 04:54:35.006955 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":42,"spinlock_wait_cycles":2816,"thread_start_us":86,"threads_started":1}
I20260808 04:54:35.007664 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.000912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:38445 (local address 127.0.0.1:45004)
0808 04:54:35.002099 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:35.002116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:35.002232 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:35.003007 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:35.003011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:35.003028 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:35.003306 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.003310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.004912 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.004915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.006149 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.006161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.007256 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.007260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.007392 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.007420 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:54:35.007470 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 04:54:35.007535 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":25,"spinlock_wait_cycles":2944,"thread_start_us":85,"threads_started":1}
I20260808 04:54:35.007710 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.001256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.15.30.65:40581)
0808 04:54:35.001585 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.001599 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.001620 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:54:35.001719 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.001724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.001963 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.002186 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:35.002516 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.003666 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.005416 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.006237 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.006348 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.006504 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.007446 (+ 942us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.007568 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":135,"threads_started":1}
I20260808 04:54:35.008828 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.001788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.15.30.67:38445)
0808 04:54:35.002520 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.002533 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.002557 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:54:35.002630 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.002635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.002832 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.003425 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:35.003560 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.004745 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.006313 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.007108 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.008206 (+ 1098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.008316 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.008355 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.008656 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":149,"threads_started":1}
I20260808 04:54:35.010326 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.000564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:39351 (local address 127.0.0.1:53720)
0808 04:54:35.000838 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:35.000856 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:35.000951 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:35.003053 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:35.003057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:35.003082 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:35.003212 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.003219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.005798 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.005803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.008041 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.008053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.009941 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.009945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.010046 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.010070 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:54:35.010122 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 04:54:35.010189 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":778,"thread_start_us":112,"threads_started":1}
I20260808 04:54:35.011021 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.000699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.15.30.66:39351)
0808 04:54:35.002520 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.002535 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.002563 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:54:35.002648 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.002655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.002873 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.003187 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:35.003390 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.005618 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.008850 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.009790 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.010582 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.010728 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.010783 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.010870 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1693,"thread_start_us":163,"threads_started":1}
I20260808 04:54:35.011229 10026 tablet_service.cc:1467] Processing CreateTablet for tablet 3c6871206cfc46d4bfb10fc4e57a8008 (DEFAULT_TABLE table=test-table [id=f8840ab731c34c65867490728d1e1ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:35.011668 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 3c6871206cfc46d4bfb10fc4e57a8008 (DEFAULT_TABLE table=test-table [id=f8840ab731c34c65867490728d1e1ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:35.012557 10026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c6871206cfc46d4bfb10fc4e57a8008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:35.012715 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c6871206cfc46d4bfb10fc4e57a8008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:35.019320 10240 tablet_service.cc:1467] Processing CreateTablet for tablet 3c6871206cfc46d4bfb10fc4e57a8008 (DEFAULT_TABLE table=test-table [id=f8840ab731c34c65867490728d1e1ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:35.020573 10240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c6871206cfc46d4bfb10fc4e57a8008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:35.022075 10614 tablet_bootstrap.cc:492] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Bootstrap starting.
I20260808 04:54:35.022653 10617 tablet_bootstrap.cc:492] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76: Bootstrap starting.
I20260808 04:54:35.024629 10614 tablet_bootstrap.cc:654] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:35.025252 10617 tablet_bootstrap.cc:654] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:35.026104 10614 log.cc:824] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:35.026168 10617 log.cc:824] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:35.028333 10614 tablet_bootstrap.cc:492] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: No bootstrap required, opened a new log
I20260808 04:54:35.028550 10614 ts_tablet_manager.cc:1397] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:54:35.029429 10617 tablet_bootstrap.cc:492] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76: No bootstrap required, opened a new log
I20260808 04:54:35.029615 10617 ts_tablet_manager.cc:1397] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:54:35.031320 10614 raft_consensus.cc:348] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.031528 10614 raft_consensus.cc:374] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:35.031560 10614 raft_consensus.cc:729] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ded059704d4f218985281966ddece5, State: Initialized, Role: FOLLOWER
I20260808 04:54:35.031994 10614 consensus_queue.cc:260] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.032457 10621 tablet_bootstrap.cc:492] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Bootstrap starting.
I20260808 04:54:35.033027 10614 ts_tablet_manager.cc:1428] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 04:54:35.033413 10586 heartbeater.cc:503] Master 127.15.30.126:34193 was elected leader, sending a full tablet report...
I20260808 04:54:35.035148 10621 tablet_bootstrap.cc:654] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:35.036067 10621 log.cc:824] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:35.038349 10617 raft_consensus.cc:348] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.038676 10617 raft_consensus.cc:374] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:35.038750 10617 raft_consensus.cc:729] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c043253988ac4e7f975b7f50f9863a76, State: Initialized, Role: FOLLOWER
I20260808 04:54:35.038848 10621 tablet_bootstrap.cc:492] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: No bootstrap required, opened a new log
I20260808 04:54:35.039028 10621 ts_tablet_manager.cc:1397] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:54:35.039211 10617 consensus_queue.cc:260] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.039851 10136 heartbeater.cc:503] Master 127.15.30.126:34193 was elected leader, sending a full tablet report...
I20260808 04:54:35.040202 10617 ts_tablet_manager.cc:1428] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260808 04:54:35.042781 10621 raft_consensus.cc:348] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.043053 10621 raft_consensus.cc:374] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:35.043119 10621 raft_consensus.cc:729] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec19f6126f054d0894dd0dff464ed85e, State: Initialized, Role: FOLLOWER
I20260808 04:54:35.043517 10621 consensus_queue.cc:260] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.044081 10354 heartbeater.cc:503] Master 127.15.30.126:34193 was elected leader, sending a full tablet report...
I20260808 04:54:35.044296 10621 ts_tablet_manager.cc:1428] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:54:35.067807 10626 raft_consensus.cc:482] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:35.068011 10626 raft_consensus.cc:504] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.069218 10626 leader_election.cc:288] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c043253988ac4e7f975b7f50f9863a76 (127.15.30.65:40581), b9ded059704d4f218985281966ddece5 (127.15.30.67:38445)
I20260808 04:54:35.072822 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.069410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:40581 (local address 127.15.30.66:34371)
0808 04:54:35.069574 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:35.069591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:35.069679 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:35.069996 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:35.069999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:35.070017 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:35.070144 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.070149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.070886 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.070892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.071695 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.071704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.072360 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.072384 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.072495 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.072522 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:54:35.072612 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:54:35.072684 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260808 04:54:35.072821 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.069651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:34371 (local address 127.15.30.65:40581)
0808 04:54:35.069793 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.069802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.069814 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:54:35.069857 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.069859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.069900 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.069987 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:35.070266 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.070775 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.071828 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.072239 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.072370 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.072454 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.072629 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.072696 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:54:35.073195 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.069584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:36009 (local address 127.15.30.67:38445)
0808 04:54:35.069721 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.069728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.069970 (+ 242us) server_negotiation.cc:407] Connection header received
0808 04:54:35.070123 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.070128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.070174 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.070256 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:35.070543 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.071032 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.072088 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.072576 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.072714 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.072764 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.073011 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.073081 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:54:35.073544 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.069439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:38445 (local address 127.15.30.66:36009)
0808 04:54:35.069859 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:35.069871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:35.070000 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:35.070279 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:35.070282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:35.070299 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:35.070427 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.070433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.071144 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.071150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.071952 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.071961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.072714 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.072718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.072833 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.072850 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 04:54:35.073343 (+ 493us) client_negotiation.cc:240] Negotiation successful
0808 04:54:35.073426 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":28,"thread_start_us":129,"threads_started":1}
I20260808 04:54:35.073523 10052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6871206cfc46d4bfb10fc4e57a8008" candidate_uuid: "ec19f6126f054d0894dd0dff464ed85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c043253988ac4e7f975b7f50f9863a76" is_pre_election: true
I20260808 04:54:35.073853 10052 raft_consensus.cc:2454] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec19f6126f054d0894dd0dff464ed85e in term 0.
I20260808 04:54:35.074198 10507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6871206cfc46d4bfb10fc4e57a8008" candidate_uuid: "ec19f6126f054d0894dd0dff464ed85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9ded059704d4f218985281966ddece5" is_pre_election: true
I20260808 04:54:35.074409 10171 leader_election.cc:302] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c043253988ac4e7f975b7f50f9863a76, ec19f6126f054d0894dd0dff464ed85e; no voters:
I20260808 04:54:35.074570 10507 raft_consensus.cc:2454] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec19f6126f054d0894dd0dff464ed85e in term 0.
I20260808 04:54:35.074667 10626 raft_consensus.cc:2781] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:54:35.074760 10626 raft_consensus.cc:482] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:54:35.074832 10626 raft_consensus.cc:3037] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:35.076027 10626 raft_consensus.cc:504] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.076427 10626 leader_election.cc:288] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [CANDIDATE]: Term 1 election: Requested vote from peers c043253988ac4e7f975b7f50f9863a76 (127.15.30.65:40581), b9ded059704d4f218985281966ddece5 (127.15.30.67:38445)
I20260808 04:54:35.076797 10052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6871206cfc46d4bfb10fc4e57a8008" candidate_uuid: "ec19f6126f054d0894dd0dff464ed85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c043253988ac4e7f975b7f50f9863a76"
I20260808 04:54:35.076836 10507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c6871206cfc46d4bfb10fc4e57a8008" candidate_uuid: "ec19f6126f054d0894dd0dff464ed85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9ded059704d4f218985281966ddece5"
I20260808 04:54:35.076910 10052 raft_consensus.cc:3037] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:35.076985 10507 raft_consensus.cc:3037] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:35.078114 10052 raft_consensus.cc:2454] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec19f6126f054d0894dd0dff464ed85e in term 1.
I20260808 04:54:35.078258 10507 raft_consensus.cc:2454] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec19f6126f054d0894dd0dff464ed85e in term 1.
I20260808 04:54:35.078486 10171 leader_election.cc:302] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c043253988ac4e7f975b7f50f9863a76, ec19f6126f054d0894dd0dff464ed85e; no voters:
I20260808 04:54:35.078647 10626 raft_consensus.cc:2781] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:35.078858 10626 raft_consensus.cc:686] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 1 LEADER]: Becoming Leader. State: Replica: ec19f6126f054d0894dd0dff464ed85e, State: Running, Role: LEADER
I20260808 04:54:35.079378 10626 consensus_queue.cc:237] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } }
I20260808 04:54:35.082561 9826 catalog_manager.cc:5495] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e reported cstate change: term changed from 0 to 1, leader changed from <none> to ec19f6126f054d0894dd0dff464ed85e (127.15.30.66). New cstate: current_term: 1 leader_uuid: "ec19f6126f054d0894dd0dff464ed85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec19f6126f054d0894dd0dff464ed85e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 } health_report { overall_health: UNKNOWN } } }
I20260808 04:54:35.109069 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.105790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.15.30.66:39351)
0808 04:54:35.105940 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.105943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.105957 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:54:35.106082 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.106084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.106140 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.106234 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:54:35.106460 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.106972 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.107781 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.107992 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.108107 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.108339 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:54:35.108751 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:54:35.108841 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.108895 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.108943 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:54:35.115180 10052 raft_consensus.cc:1261] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 1 FOLLOWER]: Refusing update from remote peer ec19f6126f054d0894dd0dff464ed85e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:35.115201 10507 raft_consensus.cc:1261] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 1 FOLLOWER]: Refusing update from remote peer ec19f6126f054d0894dd0dff464ed85e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:35.115978 10629 consensus_queue.cc:1035] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c043253988ac4e7f975b7f50f9863a76" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:35.116165 10626 consensus_queue.cc:1035] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9ded059704d4f218985281966ddece5" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:35.123986 10638 mvcc.cc:205] Tried to move back new op lower bound from 7316131328463728640 to 7316131328327536640. Current Snapshot: MvccSnapshot[applied={T|T < 7316131328463728640}]
I20260808 04:54:35.124565 10636 mvcc.cc:205] Tried to move back new op lower bound from 7316131328463728640 to 7316131328327536640. Current Snapshot: MvccSnapshot[applied={T|T < 7316131328463728640}]
I20260808 04:54:35.132679 9829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:35.132862 9829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:35.135334 9829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03070f429c3f4f77a155310e9f4245a0: Sending DeleteTablet for 3 replicas of tablet 3c6871206cfc46d4bfb10fc4e57a8008
I20260808 04:54:35.136066 10026 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c6871206cfc46d4bfb10fc4e57a8008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:52110
I20260808 04:54:35.136281 10240 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c6871206cfc46d4bfb10fc4e57a8008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:53720
I20260808 04:54:35.136281 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c6871206cfc46d4bfb10fc4e57a8008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:35 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:45004
I20260808 04:54:35.136763 10650 tablet_replica.cc:330] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: stopping tablet replica
I20260808 04:54:35.136922 10647 tablet_replica.cc:330] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76: stopping tablet replica
I20260808 04:54:35.137058 10648 tablet_replica.cc:330] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: stopping tablet replica
I20260808 04:54:35.137333 10647 raft_consensus.cc:2229] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:35.137428 10648 raft_consensus.cc:2229] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:35.137526 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 9926
I20260808 04:54:35.137815 10647 raft_consensus.cc:2258] T 3c6871206cfc46d4bfb10fc4e57a8008 P c043253988ac4e7f975b7f50f9863a76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:35.141431 10648 raft_consensus.cc:2258] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:35.144091 10650 raft_consensus.cc:2229] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:54:35.144483 10648 ts_tablet_manager.cc:1905] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:35.144657 10650 raft_consensus.cc:2258] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:35.146166 10650 ts_tablet_manager.cc:1905] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:35.148245 10648 ts_tablet_manager.cc:1918] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:35.148384 10648 log.cc:1196] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/3c6871206cfc46d4bfb10fc4e57a8008
I20260808 04:54:35.148761 10648 ts_tablet_manager.cc:1939] T 3c6871206cfc46d4bfb10fc4e57a8008 P b9ded059704d4f218985281966ddece5: Deleting consensus metadata
I20260808 04:54:35.149155 10650 ts_tablet_manager.cc:1918] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:35.149307 10650 log.cc:1196] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/3c6871206cfc46d4bfb10fc4e57a8008
I20260808 04:54:35.149677 10650 ts_tablet_manager.cc:1939] T 3c6871206cfc46d4bfb10fc4e57a8008 P ec19f6126f054d0894dd0dff464ed85e: Deleting consensus metadata
I20260808 04:54:35.150132 9804 catalog_manager.cc:4841] TS b9ded059704d4f218985281966ddece5 (127.15.30.67:38445): tablet 3c6871206cfc46d4bfb10fc4e57a8008 (table test-table [id=f8840ab731c34c65867490728d1e1ab6]) successfully deleted
I20260808 04:54:35.150870 9804 catalog_manager.cc:4841] TS ec19f6126f054d0894dd0dff464ed85e (127.15.30.66:39351): tablet 3c6871206cfc46d4bfb10fc4e57a8008 (table test-table [id=f8840ab731c34c65867490728d1e1ab6]) successfully deleted
W20260808 04:54:35.151306 9803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:40581 (error 108)
I20260808 04:54:35.151603 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 10143
I20260808 04:54:35.158491 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.158163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:40581 (local address 127.0.0.1:52118)
0808 04:54:35.158305 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:35.158388 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:40581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 04:54:35.158663 9803 catalog_manager.cc:4584] TS c043253988ac4e7f975b7f50f9863a76 (127.15.30.65:40581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c6871206cfc46d4bfb10fc4e57a8008: Network error: Client connection negotiation failed: client connection to 127.15.30.65:40581: connect: Connection refused (error 111)
I20260808 04:54:35.165609 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 10359
I20260808 04:54:35.179008 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 9781
2026-08-08T04:54:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2524 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T04:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:35Z Disabled control of system clock
I20260808 04:54:35.215337 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42787
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:36775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:54:35.326767 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:35.327013 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:35.327047 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:35.330986 10658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:35.331035 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:35.331050 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:35.331067 10658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:35.331082 10658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:35.335619 10658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:42787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:35.336584 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:35.337878 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:54:35.346485 10658 server_base.cc:1034] running on GCE node
W20260808 04:54:35.346462 10669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:35.346448 10664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:35.346449 10667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:35.347298 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:35.348353 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:35.349606 10658 hybrid_clock.cc:648] HybridClock initialized: now 1786164875349576 us; error 42 us; skew 500 ppm
I20260808 04:54:35.351405 10658 webserver.cc:460] Webserver started at http://127.15.30.126:43943/ using document root <none> and password file <none>
I20260808 04:54:35.351969 10658 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:35.352061 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:35.352304 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:35.354321 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "ccb55a5f09f04aeba6724ea91ddb80aa"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "849df3cba3f948f810f923f1df45442f"
server_key_iv: "645aa322c7a490df7222767f87ded1cd"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.354966 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "ccb55a5f09f04aeba6724ea91ddb80aa"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "849df3cba3f948f810f923f1df45442f"
server_key_iv: "645aa322c7a490df7222767f87ded1cd"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.358587 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:35.360976 10679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.362193 10658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:35.362352 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "ccb55a5f09f04aeba6724ea91ddb80aa"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "849df3cba3f948f810f923f1df45442f"
server_key_iv: "645aa322c7a490df7222767f87ded1cd"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.362466 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:35.379355 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:35.380077 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:35.380296 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:35.387476 10658 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:42787
I20260808 04:54:35.387522 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:42787 every 8 connection(s)
I20260808 04:54:35.388914 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:35.391057 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 10658
I20260808 04:54:35.391170 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:35.391477 15481 external_mini_cluster.cc:1428] Setting key aeb7d9e189d362d23ad309dbf56f6e05
I20260808 04:54:35.392524 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:35.399334 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa: Bootstrap starting.
I20260808 04:54:35.401432 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.392993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.15.30.126:42787)
0808 04:54:35.393681 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.393698 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.393739 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:54:35.394488 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.394507 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.394917 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.395225 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:35.395594 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.396745 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.398001 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.398465 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.398591 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.399508 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:35.399516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:35.399536 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:35.399913 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:35.399967 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:35.399987 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:35.400063 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.400230 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.400733 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":179,"threads_started":1}
I20260808 04:54:35.401760 10758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:35.402580 10758 log.cc:824] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:35.404585 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa: No bootstrap required, opened a new log
I20260808 04:54:35.407974 10758 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42787 } }
I20260808 04:54:35.408182 10758 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:35.408226 10758 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb55a5f09f04aeba6724ea91ddb80aa, State: Initialized, Role: FOLLOWER
I20260808 04:54:35.408646 10758 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42787 } }
I20260808 04:54:35.408775 10758 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:35.408857 10758 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:35.408990 10758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:35.409945 10758 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42787 } }
I20260808 04:54:35.410290 10758 leader_election.cc:302] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccb55a5f09f04aeba6724ea91ddb80aa; no voters:
I20260808 04:54:35.410593 10758 leader_election.cc:288] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:35.410748 10768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:35.410979 10768 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [term 1 LEADER]: Becoming Leader. State: Replica: ccb55a5f09f04aeba6724ea91ddb80aa, State: Running, Role: LEADER
I20260808 04:54:35.411259 10768 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42787 } }
I20260808 04:54:35.411814 10758 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:35.412897 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42787 } } }
I20260808 04:54:35.413038 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:35.413416 10788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:35.413415 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccb55a5f09f04aeba6724ea91ddb80aa. Latest consensus state: current_term: 1 leader_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb55a5f09f04aeba6724ea91ddb80aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 42787 } } }
I20260808 04:54:35.414206 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:35.416632 10788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:35.422675 10788 catalog_manager.cc:1347] Generated new cluster ID: dd3089c50dbe4e369a59d01711d83a31
I20260808 04:54:35.422752 10788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:35.432516 10788 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:35.433583 10788 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:35.437614 10788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa: Generated new TSK 0
I20260808 04:54:35.438220 10788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:35.444211 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42787
--builtin_ntp_servers=127.15.30.84:36775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:54:35.577167 10801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:35.577415 10801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:35.577455 10801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:35.581441 10801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:35.581489 10801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:35.581571 10801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:54:35.586172 10801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:35.587213 10801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:35.588642 10801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:35.598830 10808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:35.598888 10801 server_base.cc:1034] running on GCE node
W20260808 04:54:35.598832 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:35.598830 10806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:35.599468 10801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:35.600126 10801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:35.601320 10801 hybrid_clock.cc:648] HybridClock initialized: now 1786164875601293 us; error 38 us; skew 500 ppm
I20260808 04:54:35.603313 10801 webserver.cc:460] Webserver started at http://127.15.30.65:41681/ using document root <none> and password file <none>
I20260808 04:54:35.603916 10801 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:35.604020 10801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:35.604264 10801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:35.606376 10801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "2e243373111f48d6b1eb61ef0acc36b4"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "cd45e1f475ab313ee318f4560ca39559"
server_key_iv: "f683be0151eb2422b76853bf19311ea6"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.606971 10801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "2e243373111f48d6b1eb61ef0acc36b4"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "cd45e1f475ab313ee318f4560ca39559"
server_key_iv: "f683be0151eb2422b76853bf19311ea6"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.610677 10801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:35.613400 10820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.614639 10801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:54:35.614787 10801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "2e243373111f48d6b1eb61ef0acc36b4"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "cd45e1f475ab313ee318f4560ca39559"
server_key_iv: "f683be0151eb2422b76853bf19311ea6"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.614902 10801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:35.635365 10801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:35.636029 10801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:35.636233 10801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:35.636870 10801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:35.637847 10801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:35.637921 10801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.637995 10801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:35.638039 10801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.646855 10801 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:34769
I20260808 04:54:35.646894 11001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:34769 every 8 connection(s)
I20260808 04:54:35.647743 10801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:54:35.652644 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 10801
I20260808 04:54:35.652799 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:54:35.653098 15481 external_mini_cluster.cc:1428] Setting key e76fcbde5f811b14c932de7c2689bf73
I20260808 04:54:35.655223 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42787
--builtin_ntp_servers=127.15.30.84:36775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:54:35.657562 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.649583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:47631 (local address 127.15.30.126:42787)
0808 04:54:35.649740 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.649744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.650817 (+ 1073us) server_negotiation.cc:407] Connection header received
0808 04:54:35.652171 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.652173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.652223 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.652305 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:35.653385 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.654028 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.655420 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.655651 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.655777 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.656908 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:35.656911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:35.656914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:35.656928 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:35.656956 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:35.656965 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:35.657044 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.657239 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.657336 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:54:35.658666 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.650015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42787 (local address 127.15.30.65:47631)
0808 04:54:35.650659 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:35.650699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:35.651981 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:35.652525 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:35.652531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:35.652615 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:35.653235 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.653251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.654161 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.654164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.655222 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.655233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.655810 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.655814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.655993 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.656649 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:35.656650 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:35.656690 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:35.656751 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:35.657046 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:35.657059 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:35.657069 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 04:54:35.657224 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 04:54:35.657509 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260808 04:54:35.660005 11004 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42787
I20260808 04:54:35.660254 11004 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:35.660943 11004 heartbeater.cc:511] Master 127.15.30.126:42787 requested a full tablet report, sending...
I20260808 04:54:35.662748 10705 ts_manager.cc:194] Registered new tserver with Master: 2e243373111f48d6b1eb61ef0acc36b4 (127.15.30.65:34769)
I20260808 04:54:35.664577 10705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.65:47631
W20260808 04:54:35.769524 11008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:35.769771 11008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:35.769806 11008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:35.773720 11008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:35.773769 11008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:35.773820 11008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:54:35.778360 11008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:35.779310 11008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:35.780685 11008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:35.790333 11022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:35.790369 11024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:35.790483 11008 server_base.cc:1034] running on GCE node
W20260808 04:54:35.790279 11018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:35.790933 11008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:35.791555 11008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:35.792742 11008 hybrid_clock.cc:648] HybridClock initialized: now 1786164875792725 us; error 29 us; skew 500 ppm
I20260808 04:54:35.794662 11008 webserver.cc:460] Webserver started at http://127.15.30.66:44381/ using document root <none> and password file <none>
I20260808 04:54:35.795240 11008 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:35.795341 11008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:35.795572 11008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:35.797660 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "b064e8edd4044c0caebbc9b27973ebab"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "c5c2b84a11a59c200674f155fdce4755"
server_key_iv: "dfcfbe32b2d94f5cf502d1fe170a84cf"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.798233 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "b064e8edd4044c0caebbc9b27973ebab"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "c5c2b84a11a59c200674f155fdce4755"
server_key_iv: "dfcfbe32b2d94f5cf502d1fe170a84cf"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.801880 11008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:35.804288 11032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.805536 11008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:35.805672 11008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "b064e8edd4044c0caebbc9b27973ebab"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "c5c2b84a11a59c200674f155fdce4755"
server_key_iv: "dfcfbe32b2d94f5cf502d1fe170a84cf"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.805783 11008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:35.816215 11008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:35.816906 11008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:35.817096 11008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:35.817654 11008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:35.818634 11008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:35.818703 11008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.818771 11008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:35.818816 11008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.827651 11008 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:43559
I20260808 04:54:35.827679 11226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:43559 every 8 connection(s)
I20260808 04:54:35.828532 11008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:54:35.833336 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 11008
I20260808 04:54:35.833489 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:54:35.833775 15481 external_mini_cluster.cc:1428] Setting key efe892603b8fb60a2c5edb7fd7e46d7f
I20260808 04:54:35.835786 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42787
--builtin_ntp_servers=127.15.30.84:36775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:54:35.837967 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.830367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:49837 (local address 127.15.30.126:42787)
0808 04:54:35.830515 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:35.830519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:35.831501 (+ 982us) server_negotiation.cc:407] Connection header received
0808 04:54:35.832927 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:35.832930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:35.832997 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:35.833096 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:35.834149 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.834775 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.835934 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.836198 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.836316 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.837460 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:35.837463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:35.837465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:35.837477 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:35.837495 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:35.837503 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:35.837581 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:35.837748 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:54:35.837818 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:54:35.838821 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:35.830693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42787 (local address 127.15.30.66:49837)
0808 04:54:35.831337 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:35.831377 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:35.832699 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:35.833295 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:35.833302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:35.833386 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:35.833996 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.834012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.834905 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.834908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.835798 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:35.835808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:35.836357 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:35.836361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:35.836559 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:35.837177 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:35.837179 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:35.837218 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:35.837293 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:35.837583 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:35.837596 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:35.837604 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:54:35.837746 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 04:54:35.838025 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":122,"threads_started":1}
I20260808 04:54:35.839988 11227 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42787
I20260808 04:54:35.840219 11227 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:35.840783 11227 heartbeater.cc:511] Master 127.15.30.126:42787 requested a full tablet report, sending...
I20260808 04:54:35.841912 10705 ts_manager.cc:194] Registered new tserver with Master: b064e8edd4044c0caebbc9b27973ebab (127.15.30.66:43559)
I20260808 04:54:35.842473 10705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.66:49837
W20260808 04:54:35.950022 11234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:35.950265 11234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:35.950346 11234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:35.954365 11234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:35.954414 11234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:35.954514 11234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:54:35.959231 11234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:35.960249 11234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:35.961666 11234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:35.971190 11239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:35.971184 11244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:35.971192 11240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:35.971530 11234 server_base.cc:1034] running on GCE node
I20260808 04:54:35.971920 11234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:35.972555 11234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:35.973713 11234 hybrid_clock.cc:648] HybridClock initialized: now 1786164875973679 us; error 48 us; skew 500 ppm
I20260808 04:54:35.975450 11234 webserver.cc:460] Webserver started at http://127.15.30.67:40323/ using document root <none> and password file <none>
I20260808 04:54:35.976040 11234 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:35.976104 11234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:35.976357 11234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:35.978469 11234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "ba43783d5fcd4405a72bc11632b965d3"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "5641de690f000bcddea8b8c567659fc9"
server_key_iv: "8abe02ece063730ab1b0c3c611b4981d"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.979040 11234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "ba43783d5fcd4405a72bc11632b965d3"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "5641de690f000bcddea8b8c567659fc9"
server_key_iv: "8abe02ece063730ab1b0c3c611b4981d"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.982836 11234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:54:35.985296 11255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:35.986497 11234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:35.986617 11234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "ba43783d5fcd4405a72bc11632b965d3"
format_stamp: "Formatted at 2026-08-08 04:54:35 on dist-test-slave-zpvx"
server_key: "5641de690f000bcddea8b8c567659fc9"
server_key_iv: "8abe02ece063730ab1b0c3c611b4981d"
server_key_version: "encryptionkey@0"
I20260808 04:54:35.986723 11234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:36.018319 11234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:36.019121 11234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:36.019328 11234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:36.020004 11234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:36.021039 11234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:36.021088 11234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:36.021162 11234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:36.021203 11234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:36.030164 11234 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:44945
I20260808 04:54:36.030195 11428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:44945 every 8 connection(s)
I20260808 04:54:36.031080 11234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:54:36.033762 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 11234
I20260808 04:54:36.033888 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:54:36.034197 15481 external_mini_cluster.cc:1428] Setting key 7c6bf443252a21e7f48292ef4d4fb5e3
I20260808 04:54:36.040568 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.032963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:38423 (local address 127.15.30.126:42787)
0808 04:54:36.033096 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.033099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.034308 (+ 1209us) server_negotiation.cc:407] Connection header received
0808 04:54:36.035889 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.035892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.035949 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.036030 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:36.036986 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.037526 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.038672 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.038905 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.039021 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.040042 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:36.040046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:36.040049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:36.040062 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:36.040091 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:36.040099 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:36.040180 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.040347 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.040430 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:54:36.041424 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.033279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:42787 (local address 127.15.30.67:38423)
0808 04:54:36.034086 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:36.034142 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:36.035668 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:36.036197 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:36.036203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:36.036288 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:36.036845 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.036858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.037670 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.037674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.038540 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.038550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.039062 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.039066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.039228 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.039789 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:36.039791 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:36.039829 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:36.039891 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:36.040182 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:36.040194 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:36.040202 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:54:36.040335 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 04:54:36.040628 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260808 04:54:36.042469 11429 heartbeater.cc:347] Connected to a master server at 127.15.30.126:42787
I20260808 04:54:36.042670 11429 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:36.043107 11429 heartbeater.cc:511] Master 127.15.30.126:42787 requested a full tablet report, sending...
I20260808 04:54:36.044042 10705 ts_manager.cc:194] Registered new tserver with Master: ba43783d5fcd4405a72bc11632b965d3 (127.15.30.67:44945)
I20260808 04:54:36.044574 10705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.67:38423
I20260808 04:54:36.047961 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:54:36.053275 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.049983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.15.30.126:42787)
0808 04:54:36.050102 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.050106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.050290 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:54:36.050380 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.050382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.050419 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.050484 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:36.050735 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.051209 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.051976 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.052174 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.052247 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.052951 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:36.052954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:36.052956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:36.052968 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:36.052985 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:36.052993 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:36.053051 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.053124 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.053167 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:54:36.055941 10705 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:54:36.059024 10705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:54:36.073771 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.067878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:44945 (local address 127.0.0.1:42502)
0808 04:54:36.068581 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:36.068596 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:36.068681 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:36.069546 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:36.069551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:36.069572 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:36.069678 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.069683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.070843 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.070847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.071990 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.072001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.073174 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.073178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.073298 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.073343 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:54:36.073468 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:54:36.073593 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":86,"threads_started":1}
I20260808 04:54:36.074069 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.067551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34769 (local address 127.0.0.1:41000)
0808 04:54:36.068251 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:36.068290 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:36.068469 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:36.069891 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:36.069894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:36.069914 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:36.070003 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.070009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.071405 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.071408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.072571 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.072582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.073509 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.073513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.073602 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.073630 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:54:36.073713 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:54:36.073810 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":33,"thread_start_us":104,"threads_started":1}
I20260808 04:54:36.074198 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.068311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:43559 (local address 127.0.0.1:36962)
0808 04:54:36.068978 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:36.068993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:36.069098 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:36.069743 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:36.069747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:36.069770 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:36.069893 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.069900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.071578 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.071582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.072686 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.072700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.073786 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.073789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.073876 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.073908 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:54:36.073962 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 04:54:36.074022 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":31,"thread_start_us":119,"threads_started":1}
I20260808 04:54:36.074220 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.067713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.15.30.65:34769)
0808 04:54:36.069420 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.069433 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.069456 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:54:36.069525 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.069529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.069693 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.069904 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:36.070103 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.071277 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.072710 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.073385 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.073489 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.073661 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.073755 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.074053 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":151,"threads_started":1}
I20260808 04:54:36.074734 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.068264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.15.30.67:44945)
0808 04:54:36.068846 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.068860 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.068885 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:54:36.068965 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.068972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.069214 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.069379 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:36.069808 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.070703 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.072153 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.073041 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.073156 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.073351 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.074456 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.074556 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":178,"threads_started":1}
I20260808 04:54:36.074877 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.068770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.15.30.66:43559)
0808 04:54:36.069279 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.069292 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.069315 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:54:36.069391 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.069398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.069582 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.070039 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:36.070170 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.071279 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.072833 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.073648 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.074367 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.074557 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.074606 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.074693 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":156,"threads_started":1}
I20260808 04:54:36.077342 10897 tablet_service.cc:1467] Processing CreateTablet for tablet 245b073af1474069a6028df021d4aee3 (DEFAULT_TABLE table=test-table [id=96d0c74dd01b471fa5cb77ec407a983a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:36.077343 11113 tablet_service.cc:1467] Processing CreateTablet for tablet 245b073af1474069a6028df021d4aee3 (DEFAULT_TABLE table=test-table [id=96d0c74dd01b471fa5cb77ec407a983a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:36.078312 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245b073af1474069a6028df021d4aee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:36.078313 11113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245b073af1474069a6028df021d4aee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:36.084249 11464 tablet_bootstrap.cc:492] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Bootstrap starting.
I20260808 04:54:36.085122 11468 tablet_bootstrap.cc:492] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4: Bootstrap starting.
I20260808 04:54:36.086503 11464 tablet_bootstrap.cc:654] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:36.086604 11334 tablet_service.cc:1467] Processing CreateTablet for tablet 245b073af1474069a6028df021d4aee3 (DEFAULT_TABLE table=test-table [id=96d0c74dd01b471fa5cb77ec407a983a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:36.087221 11468 tablet_bootstrap.cc:654] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:36.087350 11464 log.cc:824] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:36.087751 11334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245b073af1474069a6028df021d4aee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:36.088049 11468 log.cc:824] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:36.089653 11464 tablet_bootstrap.cc:492] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: No bootstrap required, opened a new log
I20260808 04:54:36.089919 11464 ts_tablet_manager.cc:1397] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260808 04:54:36.090137 11468 tablet_bootstrap.cc:492] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4: No bootstrap required, opened a new log
I20260808 04:54:36.090387 11468 ts_tablet_manager.cc:1397] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260808 04:54:36.094161 11472 tablet_bootstrap.cc:492] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Bootstrap starting.
I20260808 04:54:36.094611 11464 raft_consensus.cc:348] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.094611 11468 raft_consensus.cc:348] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.094897 11468 raft_consensus.cc:374] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:36.094897 11464 raft_consensus.cc:374] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:36.094967 11468 raft_consensus.cc:729] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e243373111f48d6b1eb61ef0acc36b4, State: Initialized, Role: FOLLOWER
I20260808 04:54:36.094976 11464 raft_consensus.cc:729] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b064e8edd4044c0caebbc9b27973ebab, State: Initialized, Role: FOLLOWER
I20260808 04:54:36.095456 11464 consensus_queue.cc:260] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.095456 11468 consensus_queue.cc:260] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.096335 11004 heartbeater.cc:503] Master 127.15.30.126:42787 was elected leader, sending a full tablet report...
I20260808 04:54:36.096527 11468 ts_tablet_manager.cc:1428] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:54:36.096527 11464 ts_tablet_manager.cc:1428] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260808 04:54:36.096989 11227 heartbeater.cc:503] Master 127.15.30.126:42787 was elected leader, sending a full tablet report...
I20260808 04:54:36.097383 11472 tablet_bootstrap.cc:654] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:36.098300 11472 log.cc:824] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:36.100235 11472 tablet_bootstrap.cc:492] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: No bootstrap required, opened a new log
I20260808 04:54:36.100481 11472 ts_tablet_manager.cc:1397] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:54:36.103636 11472 raft_consensus.cc:348] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.103886 11472 raft_consensus.cc:374] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:36.103971 11472 raft_consensus.cc:729] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba43783d5fcd4405a72bc11632b965d3, State: Initialized, Role: FOLLOWER
I20260808 04:54:36.104444 11472 consensus_queue.cc:260] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.105120 11429 heartbeater.cc:503] Master 127.15.30.126:42787 was elected leader, sending a full tablet report...
I20260808 04:54:36.105535 11472 ts_tablet_manager.cc:1428] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260808 04:54:36.148932 11005 tablet.cc:2369] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:36.180935 11473 raft_consensus.cc:482] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:36.181133 11473 raft_consensus.cc:504] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.182222 11473 leader_election.cc:288] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b064e8edd4044c0caebbc9b27973ebab (127.15.30.66:43559), ba43783d5fcd4405a72bc11632b965d3 (127.15.30.67:44945)
I20260808 04:54:36.187770 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.182599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:58513 (local address 127.15.30.67:44945)
0808 04:54:36.182739 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.182744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.183161 (+ 417us) server_negotiation.cc:407] Connection header received
0808 04:54:36.183249 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.183254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.183305 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.183378 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:36.183673 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.184506 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.186459 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.187065 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.187202 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.187256 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.187550 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.187641 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:54:36.187790 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.182599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:35101 (local address 127.15.30.66:43559)
0808 04:54:36.182750 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.182755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.182926 (+ 171us) server_negotiation.cc:407] Connection header received
0808 04:54:36.183103 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.183112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.183192 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.183281 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:36.183692 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.184516 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.186511 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.187095 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.187202 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.187273 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.187602 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.187679 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:54:36.187800 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.182481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:43559 (local address 127.15.30.65:35101)
0808 04:54:36.182829 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:36.182842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:36.182980 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:36.183284 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:36.183286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:36.183305 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:36.183521 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.183530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.184625 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.184628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.186263 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.186274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.187215 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.187219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.187392 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.187450 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 04:54:36.187560 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:54:36.187609 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":268,"thread_start_us":106,"threads_started":1}
I20260808 04:54:36.187803 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.182481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:44945 (local address 127.15.30.65:58513)
0808 04:54:36.183061 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:36.183073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:36.183163 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:36.183413 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:36.183416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:36.183430 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:36.183558 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.183567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.184625 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.184628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.186261 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:36.186274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.187215 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.187219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:36.187366 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.187444 (+ 78us) client_negotiation.cc:769] Sending connection context
0808 04:54:36.187528 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:54:36.187608 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":487,"thread_start_us":97,"threads_started":1}
I20260808 04:54:36.188504 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245b073af1474069a6028df021d4aee3" candidate_uuid: "2e243373111f48d6b1eb61ef0acc36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b064e8edd4044c0caebbc9b27973ebab" is_pre_election: true
I20260808 04:54:36.188689 11360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245b073af1474069a6028df021d4aee3" candidate_uuid: "2e243373111f48d6b1eb61ef0acc36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba43783d5fcd4405a72bc11632b965d3" is_pre_election: true
I20260808 04:54:36.188836 11149 raft_consensus.cc:2454] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e243373111f48d6b1eb61ef0acc36b4 in term 0.
I20260808 04:54:36.188946 11360 raft_consensus.cc:2454] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e243373111f48d6b1eb61ef0acc36b4 in term 0.
I20260808 04:54:36.189350 10823 leader_election.cc:302] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e243373111f48d6b1eb61ef0acc36b4, b064e8edd4044c0caebbc9b27973ebab, ba43783d5fcd4405a72bc11632b965d3; no voters:
I20260808 04:54:36.189625 11473 raft_consensus.cc:2781] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:54:36.189733 11473 raft_consensus.cc:482] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:54:36.189795 11473 raft_consensus.cc:3037] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:36.190927 11473 raft_consensus.cc:504] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.191279 11473 leader_election.cc:288] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [CANDIDATE]: Term 1 election: Requested vote from peers b064e8edd4044c0caebbc9b27973ebab (127.15.30.66:43559), ba43783d5fcd4405a72bc11632b965d3 (127.15.30.67:44945)
I20260808 04:54:36.191632 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245b073af1474069a6028df021d4aee3" candidate_uuid: "2e243373111f48d6b1eb61ef0acc36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b064e8edd4044c0caebbc9b27973ebab"
I20260808 04:54:36.191632 11360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245b073af1474069a6028df021d4aee3" candidate_uuid: "2e243373111f48d6b1eb61ef0acc36b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba43783d5fcd4405a72bc11632b965d3"
I20260808 04:54:36.191756 11360 raft_consensus.cc:3037] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:36.191756 11149 raft_consensus.cc:3037] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:36.193070 11149 raft_consensus.cc:2454] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e243373111f48d6b1eb61ef0acc36b4 in term 1.
I20260808 04:54:36.193070 11360 raft_consensus.cc:2454] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e243373111f48d6b1eb61ef0acc36b4 in term 1.
I20260808 04:54:36.193429 10827 leader_election.cc:302] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e243373111f48d6b1eb61ef0acc36b4, b064e8edd4044c0caebbc9b27973ebab, ba43783d5fcd4405a72bc11632b965d3; no voters:
I20260808 04:54:36.193655 11473 raft_consensus.cc:2781] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:36.193820 11473 raft_consensus.cc:686] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2e243373111f48d6b1eb61ef0acc36b4, State: Running, Role: LEADER
I20260808 04:54:36.194140 11473 consensus_queue.cc:237] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } }
I20260808 04:54:36.197228 10705 catalog_manager.cc:5495] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e243373111f48d6b1eb61ef0acc36b4 (127.15.30.65). New cstate: current_term: 1 leader_uuid: "2e243373111f48d6b1eb61ef0acc36b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e243373111f48d6b1eb61ef0acc36b4" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 } health_report { overall_health: UNKNOWN } } }
I20260808 04:54:36.224866 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.221261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.15.30.65:34769)
0808 04:54:36.221420 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:36.221425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:36.221439 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:54:36.221606 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:36.221609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:36.221662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:36.221774 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:54:36.222046 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.222544 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.223397 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:36.223612 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:36.223719 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:36.224033 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:54:36.224480 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:54:36.224611 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:36.224676 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 04:54:36.224726 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:54:36.230839 11149 raft_consensus.cc:1261] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 1 FOLLOWER]: Refusing update from remote peer 2e243373111f48d6b1eb61ef0acc36b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:36.230840 11360 raft_consensus.cc:1261] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 1 FOLLOWER]: Refusing update from remote peer 2e243373111f48d6b1eb61ef0acc36b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:36.231634 11484 consensus_queue.cc:1035] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba43783d5fcd4405a72bc11632b965d3" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:36.231822 11473 consensus_queue.cc:1035] T 245b073af1474069a6028df021d4aee3 P 2e243373111f48d6b1eb61ef0acc36b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b064e8edd4044c0caebbc9b27973ebab" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:36.248440 10705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:36.248597 10705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:36.251057 10705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccb55a5f09f04aeba6724ea91ddb80aa: Sending DeleteTablet for 3 replicas of tablet 245b073af1474069a6028df021d4aee3
I20260808 04:54:36.251909 11334 tablet_service.cc:1514] Processing DeleteTablet for tablet 245b073af1474069a6028df021d4aee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:36 UTC) from {username='slave'} at 127.0.0.1:42502
I20260808 04:54:36.252043 10897 tablet_service.cc:1514] Processing DeleteTablet for tablet 245b073af1474069a6028df021d4aee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:36 UTC) from {username='slave'} at 127.0.0.1:41000
I20260808 04:54:36.252497 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 10801
I20260808 04:54:36.252506 11113 tablet_service.cc:1514] Processing DeleteTablet for tablet 245b073af1474069a6028df021d4aee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:36 UTC) from {username='slave'} at 127.0.0.1:36962
I20260808 04:54:36.259757 11514 tablet_replica.cc:330] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: stopping tablet replica
I20260808 04:54:36.260095 11514 raft_consensus.cc:2229] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:36.260418 11514 raft_consensus.cc:2258] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:36.262845 11514 ts_tablet_manager.cc:1905] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:36.264621 11510 tablet_replica.cc:330] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: stopping tablet replica
I20260808 04:54:36.264961 11510 raft_consensus.cc:2229] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:36.265317 11510 raft_consensus.cc:2258] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:36.265393 11514 ts_tablet_manager.cc:1918] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:36.265494 11514 log.cc:1196] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/245b073af1474069a6028df021d4aee3
I20260808 04:54:36.265843 11514 ts_tablet_manager.cc:1939] T 245b073af1474069a6028df021d4aee3 P b064e8edd4044c0caebbc9b27973ebab: Deleting consensus metadata
I20260808 04:54:36.266943 11510 ts_tablet_manager.cc:1905] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:36.267194 10682 catalog_manager.cc:4841] TS b064e8edd4044c0caebbc9b27973ebab (127.15.30.66:43559): tablet 245b073af1474069a6028df021d4aee3 (table test-table [id=96d0c74dd01b471fa5cb77ec407a983a]) successfully deleted
I20260808 04:54:36.268702 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 11008
W20260808 04:54:36.268721 10682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:34769 (error 108)
I20260808 04:54:36.273862 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:36.273565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34769 (local address 127.0.0.1:41016)
0808 04:54:36.273698 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:36.273782 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 04:54:36.274037 10682 catalog_manager.cc:4584] TS 2e243373111f48d6b1eb61ef0acc36b4 (127.15.30.65:34769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 245b073af1474069a6028df021d4aee3: Network error: Client connection negotiation failed: client connection to 127.15.30.65:34769: connect: Connection refused (error 111)
I20260808 04:54:36.274349 11510 ts_tablet_manager.cc:1918] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:36.274437 11510 log.cc:1196] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/245b073af1474069a6028df021d4aee3
I20260808 04:54:36.274682 11510 ts_tablet_manager.cc:1939] T 245b073af1474069a6028df021d4aee3 P ba43783d5fcd4405a72bc11632b965d3: Deleting consensus metadata
I20260808 04:54:36.276544 10686 catalog_manager.cc:4841] TS ba43783d5fcd4405a72bc11632b965d3 (127.15.30.67:44945): tablet 245b073af1474069a6028df021d4aee3 (table test-table [id=96d0c74dd01b471fa5cb77ec407a983a]) successfully deleted
I20260808 04:54:36.282491 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 11234
I20260808 04:54:36.299258 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 10658
2026-08-08T04:54:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1122 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:36 dist-test-slave-zpvx krb5kdc[11520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:36 dist-test-slave-zpvx krb5kdc[11520](info): set up 2 sockets
Aug 08 04:54:36 dist-test-slave-zpvx krb5kdc[11520](info): commencing operation
krb5kdc: starting...
W20260808 04:54:37.569960 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:37 dist-test-slave-zpvx krb5kdc[11520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:54:37.695775 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43731
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:37.804126 11536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:37.804453 11536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:37.804510 11536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:37.808357 11536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:37.808420 11536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:37.808439 11536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:37.808455 11536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:37.808470 11536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:37.813539 11536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:43731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:37.814477 11536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:37.815433 11536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:37.823746 11547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:37.823781 11546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:37.823772 11549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:37.823956 11536 server_base.cc:1034] running on GCE node
I20260808 04:54:37.824538 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:37.825505 11536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:37.826732 11536 hybrid_clock.cc:648] HybridClock initialized: now 1786164877826719 us; error 25 us; skew 500 ppm
Aug 08 04:54:37 dist-test-slave-zpvx krb5kdc[11520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164877, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:37.836205 11536 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:54:37.837113 11536 webserver.cc:460] Webserver started at http://127.15.30.126:43273/ using document root <none> and password file <none>
I20260808 04:54:37.837687 11536 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:37.837781 11536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:37.838014 11536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:37.840059 11536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "0ff0c857c79f429d9636ad8a0a727369"
format_stamp: "Formatted at 2026-08-08 04:54:37 on dist-test-slave-zpvx"
server_key: "fe3cd174af19aaccfe22172599a90a91"
server_key_iv: "801215c753b81d322c32fd4af8af4ee8"
server_key_version: "encryptionkey@0"
I20260808 04:54:37.840654 11536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "0ff0c857c79f429d9636ad8a0a727369"
format_stamp: "Formatted at 2026-08-08 04:54:37 on dist-test-slave-zpvx"
server_key: "fe3cd174af19aaccfe22172599a90a91"
server_key_iv: "801215c753b81d322c32fd4af8af4ee8"
server_key_version: "encryptionkey@0"
I20260808 04:54:37.844344 11536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:54:37.846807 11558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:37.848008 11536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:37.848184 11536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "0ff0c857c79f429d9636ad8a0a727369"
format_stamp: "Formatted at 2026-08-08 04:54:37 on dist-test-slave-zpvx"
server_key: "fe3cd174af19aaccfe22172599a90a91"
server_key_iv: "801215c753b81d322c32fd4af8af4ee8"
server_key_version: "encryptionkey@0"
I20260808 04:54:37.848309 11536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:37.878851 11536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:37.881688 11536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:37.881896 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:37.889150 11536 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:43731
I20260808 04:54:37.889180 11654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:43731 every 8 connection(s)
I20260808 04:54:37.890061 11536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:37.892017 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 11536
I20260808 04:54:37.892165 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:37.892484 15481 external_mini_cluster.cc:1428] Setting key d416fb5e853380e6d4083d0fb38320bb
I20260808 04:54:37.893082 11655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:37.898903 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369: Bootstrap starting.
Aug 08 04:54:37 dist-test-slave-zpvx krb5kdc[11520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:37.901670 11655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:37.902397 11655 log.cc:824] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:37.904552 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369: No bootstrap required, opened a new log
I20260808 04:54:37.907014 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:37.893916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.15.30.126:43731)
0808 04:54:37.894356 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:37.894371 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:37.894406 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:54:37.895050 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:37.895066 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:37.895381 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:37.895716 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:37.896592 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:37.897589 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:37.898477 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:37.898959 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:37.899149 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:37.901876 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:37.901881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:37.901893 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:37.901920 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:37.903920 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:37.904357 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:37.904363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:37.904392 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:37.904530 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:37.904837 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:37.904840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:37.904842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:37.905348 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:37.905541 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:37.905742 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 04:54:37.906029 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260808 04:54:37.909096 11655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff0c857c79f429d9636ad8a0a727369" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43731 } }
I20260808 04:54:37.909368 11655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:37.909435 11655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff0c857c79f429d9636ad8a0a727369, State: Initialized, Role: FOLLOWER
I20260808 04:54:37.909847 11655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff0c857c79f429d9636ad8a0a727369" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43731 } }
I20260808 04:54:37.909991 11655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:37.910056 11655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:37.910161 11655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:37.911042 11655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff0c857c79f429d9636ad8a0a727369" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43731 } }
I20260808 04:54:37.911401 11655 leader_election.cc:302] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ff0c857c79f429d9636ad8a0a727369; no voters:
I20260808 04:54:37.911713 11655 leader_election.cc:288] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:37.911857 11665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:37.912109 11665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [term 1 LEADER]: Becoming Leader. State: Replica: 0ff0c857c79f429d9636ad8a0a727369, State: Running, Role: LEADER
I20260808 04:54:37.912433 11665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff0c857c79f429d9636ad8a0a727369" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43731 } }
I20260808 04:54:37.912956 11655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:37.914013 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ff0c857c79f429d9636ad8a0a727369. Latest consensus state: current_term: 1 leader_uuid: "0ff0c857c79f429d9636ad8a0a727369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff0c857c79f429d9636ad8a0a727369" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43731 } } }
I20260808 04:54:37.914158 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:37.914386 11666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ff0c857c79f429d9636ad8a0a727369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ff0c857c79f429d9636ad8a0a727369" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43731 } } }
I20260808 04:54:37.914484 11666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:37.914901 11678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:37.917668 11678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:37.922477 11678 catalog_manager.cc:1347] Generated new cluster ID: 6eaf93ca98be4f36ac1f3a1356fef9fa
I20260808 04:54:37.922539 11678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:37.937093 11678 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:37.937947 11678 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:37.949944 11678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369: Generated new TSK 0
I20260808 04:54:37.950815 11678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:54:37.989408 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:43731
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:38.125310 11699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:38.125602 11699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:38.125671 11699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:38.129588 11699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:38.129633 11699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:38.129688 11699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:54:38.134647 11699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:38.135586 11699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:38.136624 11699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:38.146301 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:38.146330 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:38.146320 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:38.146649 11699 server_base.cc:1034] running on GCE node
I20260808 04:54:38.147035 11699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:38.147740 11699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:38.148905 11699 hybrid_clock.cc:648] HybridClock initialized: now 1786164878148893 us; error 25 us; skew 500 ppm
Aug 08 04:54:38 dist-test-slave-zpvx krb5kdc[11520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:38.151887 11699 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:54:38.152817 11699 webserver.cc:460] Webserver started at http://127.15.30.65:42685/ using document root <none> and password file <none>
I20260808 04:54:38.153393 11699 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:38.153489 11699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:38.153728 11699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:38.155773 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "c8f21a8246f940ebb0b42c1aadf854f2"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "4884a50f063234b121df06457b14a332"
server_key_iv: "98bd46d07000a451e7d771524167aa65"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.156405 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "c8f21a8246f940ebb0b42c1aadf854f2"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "4884a50f063234b121df06457b14a332"
server_key_iv: "98bd46d07000a451e7d771524167aa65"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.160005 11699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:54:38.162514 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.163697 11699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:38.163838 11699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "c8f21a8246f940ebb0b42c1aadf854f2"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "4884a50f063234b121df06457b14a332"
server_key_iv: "98bd46d07000a451e7d771524167aa65"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.163949 11699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:38.179818 11699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:38.182993 11699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:38.183199 11699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:38.183791 11699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:38.184660 11699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:38.184732 11699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.184800 11699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:38.184854 11699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.193565 11699 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:34003
I20260808 04:54:38.193609 11899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:34003 every 8 connection(s)
I20260808 04:54:38.194432 11699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:54:38.196682 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 11699
I20260808 04:54:38.196828 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:54:38.197117 15481 external_mini_cluster.cc:1428] Setting key 62ae8f252c181e9b0bf52c6f513e8918
Aug 08 04:54:38 dist-test-slave-zpvx krb5kdc[11520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:54:38.209038 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.196149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:45171 (local address 127.15.30.126:43731)
0808 04:54:38.196308 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.196313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.197496 (+ 1183us) server_negotiation.cc:407] Connection header received
0808 04:54:38.198472 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.198475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.198524 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.198622 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:38.200216 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.200871 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.202045 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.202317 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.202401 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.205248 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:38.205251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:38.205254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:38.205287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:38.206743 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.207388 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.207390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.207391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.207467 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.207849 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.207851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.207852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.208020 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:38.208097 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.208766 (+ 669us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.208873 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:54:38.210016 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.196602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:43731 (local address 127.15.30.65:45171)
0808 04:54:38.197322 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.197376 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.198226 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.198834 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.198842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.199356 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:38.200049 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.200072 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.201041 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.201045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.201914 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.201924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.202454 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.202457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.202595 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.203185 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:38.203201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:38.205070 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:38.206868 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:38.206874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:38.206889 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:38.207276 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:38.207561 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:38.207565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:38.207567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:38.207746 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:38.208136 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:38.208146 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:38.208511 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.208728 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.209086 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":144,"threads_started":1}
I20260808 04:54:38.211213 11901 heartbeater.cc:347] Connected to a master server at 127.15.30.126:43731
I20260808 04:54:38.211491 11901 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:54:38.212078 11901 heartbeater.cc:511] Master 127.15.30.126:43731 requested a full tablet report, sending...
I20260808 04:54:38.213797 11588 ts_manager.cc:194] Registered new tserver with Master: c8f21a8246f940ebb0b42c1aadf854f2 (127.15.30.65:34003)
I20260808 04:54:38.215021 11588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:45171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:54:38.227085 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:43731
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:38.336774 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:38.337013 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:38.337054 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:38.340994 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:38.341043 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:38.341144 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:54:38.346194 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:38.347177 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:38.348176 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:38.357630 11926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:38.357689 11912 server_base.cc:1034] running on GCE node
W20260808 04:54:38.357616 11931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:38.357653 11927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:38.358217 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:38.358865 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:38.360057 11912 hybrid_clock.cc:648] HybridClock initialized: now 1786164878360027 us; error 49 us; skew 500 ppm
Aug 08 04:54:38 dist-test-slave-zpvx krb5kdc[11520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:38.362828 11912 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:54:38.363627 11912 webserver.cc:460] Webserver started at http://127.15.30.66:33839/ using document root <none> and password file <none>
I20260808 04:54:38.364182 11912 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:38.364276 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:38.364554 11912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:38.366557 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "61c43b61fb5948b9b746cbad21e6a286"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "a3e3dcbefd79051a9761ab01f3074354"
server_key_iv: "d5fe56ad755ddf526fea71149ba42e70"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.367142 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "61c43b61fb5948b9b746cbad21e6a286"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "a3e3dcbefd79051a9761ab01f3074354"
server_key_iv: "d5fe56ad755ddf526fea71149ba42e70"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.370801 11912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:38.373045 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.374220 11912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:54:38.374368 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "61c43b61fb5948b9b746cbad21e6a286"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "a3e3dcbefd79051a9761ab01f3074354"
server_key_iv: "d5fe56ad755ddf526fea71149ba42e70"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.374476 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:38.406323 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:38.409220 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:38.409425 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:38.410014 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:38.410848 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:38.410933 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.411024 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:38.411075 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.420262 11912 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:36761
I20260808 04:54:38.420290 12131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:36761 every 8 connection(s)
I20260808 04:54:38.421221 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:54:38.423305 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 11912
I20260808 04:54:38.423432 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:54:38.423797 15481 external_mini_cluster.cc:1428] Setting key 89c9f694d7532f30bd4b812bd92d697e
Aug 08 04:54:38 dist-test-slave-zpvx krb5kdc[11520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
I20260808 04:54:38.435217 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.422989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:51189 (local address 127.15.30.126:43731)
0808 04:54:38.423130 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.423134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.424963 (+ 1829us) server_negotiation.cc:407] Connection header received
0808 04:54:38.425838 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.425841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.425894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.425973 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:38.427072 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.427649 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.428918 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.429167 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.429282 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.431653 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:38.431657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:38.431660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:38.431688 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:38.433175 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.433690 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.433694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.433695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.433778 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.434130 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.434135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.434137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.434314 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:38.434425 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.434920 (+ 495us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.435044 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:54:38.436012 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.423427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:43731 (local address 127.15.30.66:51189)
0808 04:54:38.424821 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.424862 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.425643 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.426141 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.426148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.426478 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:38.426923 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.426936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.427782 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.427786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.428794 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.428804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.429323 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.429328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.429488 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.430070 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:38.430086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:38.431492 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:38.433300 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:38.433305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:38.433319 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:38.433579 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:38.433878 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:38.433881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:38.433883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:38.434022 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:38.434425 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:38.434432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:38.434663 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.434864 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.435196 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
I20260808 04:54:38.437225 12132 heartbeater.cc:347] Connected to a master server at 127.15.30.126:43731
I20260808 04:54:38.437472 12132 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:38.437983 12132 heartbeater.cc:511] Master 127.15.30.126:43731 requested a full tablet report, sending...
I20260808 04:54:38.439179 11588 ts_manager.cc:194] Registered new tserver with Master: 61c43b61fb5948b9b746cbad21e6a286 (127.15.30.66:36761)
I20260808 04:54:38.439779 11588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:51189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:54:38.452126 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:43731
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:38.565594 12139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:38.565845 12139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:38.565913 12139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:38.570137 12139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:38.570216 12139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:38.570297 12139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:54:38.575428 12139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:43731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:38.576431 12139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:38.577404 12139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:38.586709 12146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:38.586772 12148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:38.586781 12153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:38.586943 12139 server_base.cc:1034] running on GCE node
I20260808 04:54:38.587347 12139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:38.588027 12139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:38.589210 12139 hybrid_clock.cc:648] HybridClock initialized: now 1786164878589193 us; error 38 us; skew 500 ppm
Aug 08 04:54:38 dist-test-slave-zpvx krb5kdc[11520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:38.591917 12139 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:54:38.592736 12139 webserver.cc:460] Webserver started at http://127.15.30.67:32899/ using document root <none> and password file <none>
I20260808 04:54:38.593228 12139 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:38.593286 12139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:38.593436 12139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:38.595304 12139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "2480d0d939f745e8b72f8b8f76bb8da2"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "a1aec87e0426969c4f0e83ada9233925"
server_key_iv: "b6be2a2e0aa0968637574e50481bfa48"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.595872 12139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "2480d0d939f745e8b72f8b8f76bb8da2"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "a1aec87e0426969c4f0e83ada9233925"
server_key_iv: "b6be2a2e0aa0968637574e50481bfa48"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.599411 12139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:54:38.601824 12162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.602985 12139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:54:38.603107 12139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "2480d0d939f745e8b72f8b8f76bb8da2"
format_stamp: "Formatted at 2026-08-08 04:54:38 on dist-test-slave-zpvx"
server_key: "a1aec87e0426969c4f0e83ada9233925"
server_key_iv: "b6be2a2e0aa0968637574e50481bfa48"
server_key_version: "encryptionkey@0"
I20260808 04:54:38.603206 12139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:38.617803 12139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:38.621078 12139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:38.621280 12139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:38.621869 12139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:38.622665 12139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:38.622711 12139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.622781 12139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:38.622823 12139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:38.631680 12139 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:46715
I20260808 04:54:38.631711 12340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:46715 every 8 connection(s)
I20260808 04:54:38.632529 12139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:54:38.638558 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 12139
I20260808 04:54:38.638697 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:54:38.639004 15481 external_mini_cluster.cc:1428] Setting key 8b84e2542e0cbcb66524a9878309130f
Aug 08 04:54:38 dist-test-slave-zpvx krb5kdc[11520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:38.645707 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.634268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:34533 (local address 127.15.30.126:43731)
0808 04:54:38.634410 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.634416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.635390 (+ 974us) server_negotiation.cc:407] Connection header received
0808 04:54:38.636266 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.636269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.636327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.636467 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:38.637622 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.638205 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.639570 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.639871 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.639983 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.642540 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:38.642543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:38.642545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:38.642570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:38.643966 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.644457 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.644460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.644461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.644535 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.644830 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.644832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.644833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.644995 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:38.645069 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.645501 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.645593 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:54:38.646474 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.634610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:43731 (local address 127.15.30.67:34533)
0808 04:54:38.635229 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.635277 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.636067 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.636668 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.636675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.637005 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:38.637475 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.637488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.638342 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.638346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.639439 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.639454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.640032 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.640037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.640186 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.640814 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:38.640835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:38.642393 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:38.644062 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:38.644066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:38.644076 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:38.644321 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:38.644615 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:38.644618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:38.644619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:38.644748 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:38.645083 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:38.645088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:38.645302 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.645452 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.645671 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":117,"threads_started":1}
I20260808 04:54:38.647568 12341 heartbeater.cc:347] Connected to a master server at 127.15.30.126:43731
I20260808 04:54:38.647809 12341 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:38.648280 12341 heartbeater.cc:511] Master 127.15.30.126:43731 requested a full tablet report, sending...
I20260808 04:54:38.649361 11588 ts_manager.cc:194] Registered new tserver with Master: 2480d0d939f745e8b72f8b8f76bb8da2 (127.15.30.67:46715)
I20260808 04:54:38.649931 11588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:34533
I20260808 04:54:38.653053 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:54:38.662319 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.655030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.15.30.126:43731)
0808 04:54:38.655159 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.655162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.655266 (+ 104us) server_negotiation.cc:407] Connection header received
0808 04:54:38.655353 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.655355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.655388 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.655457 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:38.656114 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.656685 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.657484 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.657694 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.657771 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.659272 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:38.659276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:38.659278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:38.659300 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:38.660584 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.661035 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.661037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.661038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.661111 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:38.661378 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:38.661380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:38.661381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:38.661573 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:38.661688 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.662065 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.662173 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:54:38.665242 11588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:54:38.668341 11588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:54:38.683634 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.676661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34003 (local address 127.0.0.1:60666)
0808 04:54:38.677420 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.677473 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.677600 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.678702 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.678708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.678723 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:38.678873 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.678878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.680451 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.680454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.681982 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.681993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.683081 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.683084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.683200 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.683248 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.683360 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.683479 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":34,"spinlock_wait_cycles":6528,"thread_start_us":102,"threads_started":1}
I20260808 04:54:38.683667 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.676865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.15.30.65:34003)
0808 04:54:38.678176 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.678195 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.678224 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:54:38.678307 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.678315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.678535 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.679043 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:38.679200 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.680282 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.682116 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.682948 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.683076 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.683248 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.683391 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.683488 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":98,"threads_started":1}
I20260808 04:54:38.684185 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.677243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.15.30.67:46715)
0808 04:54:38.678001 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.678013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.678197 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:54:38.678307 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.678312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.678532 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.678691 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:38.679027 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.680239 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.682116 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.682948 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.683076 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.683248 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.683907 (+ 659us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.684032 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":165,"threads_started":1}
I20260808 04:54:38.684404 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.678099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:36761 (local address 127.0.0.1:39696)
0808 04:54:38.678527 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.678541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.678633 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.679362 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.679365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.679383 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:38.679480 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.679487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.680534 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.680537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.682728 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.682735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.683959 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.683963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.684070 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.684100 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.684172 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.684244 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":46,"thread_start_us":97,"threads_started":1}
I20260808 04:54:38.684809 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.678242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.15.30.66:36761)
0808 04:54:38.679009 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.679020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.679040 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:54:38.679107 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.679110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.679228 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.679381 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:38.679604 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.680352 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.682848 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.683624 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.683733 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.683898 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.684570 (+ 672us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.684665 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":162,"threads_started":1}
I20260808 04:54:38.686728 11806 tablet_service.cc:1467] Processing CreateTablet for tablet f128b012ebf248a6aed892a3fcce7249 (DEFAULT_TABLE table=test-table [id=e23c6b7397c54d53b4a5e965806ecfb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:38.687661 11806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f128b012ebf248a6aed892a3fcce7249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:38.683817 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.676857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:46715 (local address 127.0.0.1:51442)
0808 04:54:38.677454 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.677477 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.677595 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.678679 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.678684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.678706 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:38.678880 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.678890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.680752 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.680755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.681983 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.681993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.683337 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.683351 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.683460 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.683484 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.683591 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.683658 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":44,"spinlock_wait_cycles":2048,"thread_start_us":101,"threads_started":1}
I20260808 04:54:38.691262 12020 tablet_service.cc:1467] Processing CreateTablet for tablet f128b012ebf248a6aed892a3fcce7249 (DEFAULT_TABLE table=test-table [id=e23c6b7397c54d53b4a5e965806ecfb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:38.692297 12020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f128b012ebf248a6aed892a3fcce7249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:38.692811 12229 tablet_service.cc:1467] Processing CreateTablet for tablet f128b012ebf248a6aed892a3fcce7249 (DEFAULT_TABLE table=test-table [id=e23c6b7397c54d53b4a5e965806ecfb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:38.693848 12229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f128b012ebf248a6aed892a3fcce7249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:38.696784 12362 tablet_bootstrap.cc:492] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2: Bootstrap starting.
I20260808 04:54:38.698657 12363 tablet_bootstrap.cc:492] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Bootstrap starting.
I20260808 04:54:38.699174 12362 tablet_bootstrap.cc:654] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:38.700533 12362 log.cc:824] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:38.701001 12363 tablet_bootstrap.cc:654] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:38.701071 12364 tablet_bootstrap.cc:492] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Bootstrap starting.
I20260808 04:54:38.701835 12363 log.cc:824] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:38.703147 12364 tablet_bootstrap.cc:654] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:38.703589 12363 tablet_bootstrap.cc:492] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: No bootstrap required, opened a new log
I20260808 04:54:38.703691 12362 tablet_bootstrap.cc:492] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2: No bootstrap required, opened a new log
I20260808 04:54:38.703806 12363 ts_tablet_manager.cc:1397] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260808 04:54:38.703895 12362 ts_tablet_manager.cc:1397] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260808 04:54:38.704329 12364 log.cc:824] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:38.706193 12364 tablet_bootstrap.cc:492] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: No bootstrap required, opened a new log
I20260808 04:54:38.706380 12364 ts_tablet_manager.cc:1397] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:54:38.707130 12363 raft_consensus.cc:348] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.707370 12363 raft_consensus.cc:374] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:38.707425 12363 raft_consensus.cc:729] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c43b61fb5948b9b746cbad21e6a286, State: Initialized, Role: FOLLOWER
I20260808 04:54:38.707561 12362 raft_consensus.cc:348] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.707818 12362 raft_consensus.cc:374] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:38.707875 12362 raft_consensus.cc:729] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f21a8246f940ebb0b42c1aadf854f2, State: Initialized, Role: FOLLOWER
I20260808 04:54:38.707998 12363 consensus_queue.cc:260] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.708294 12362 consensus_queue.cc:260] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.709100 12363 ts_tablet_manager.cc:1428] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260808 04:54:38.709239 12362 ts_tablet_manager.cc:1428] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:54:38.709460 12132 heartbeater.cc:503] Master 127.15.30.126:43731 was elected leader, sending a full tablet report...
I20260808 04:54:38.709604 11901 heartbeater.cc:503] Master 127.15.30.126:43731 was elected leader, sending a full tablet report...
I20260808 04:54:38.711050 12364 raft_consensus.cc:348] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.711293 12364 raft_consensus.cc:374] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:38.711351 12364 raft_consensus.cc:729] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2480d0d939f745e8b72f8b8f76bb8da2, State: Initialized, Role: FOLLOWER
I20260808 04:54:38.711735 12364 consensus_queue.cc:260] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.712522 12341 heartbeater.cc:503] Master 127.15.30.126:43731 was elected leader, sending a full tablet report...
I20260808 04:54:38.712663 12364 ts_tablet_manager.cc:1428] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:54:38.860432 12373 raft_consensus.cc:482] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:38.860674 12373 raft_consensus.cc:504] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.861855 12373 leader_election.cc:288] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8f21a8246f940ebb0b42c1aadf854f2 (127.15.30.65:34003), 61c43b61fb5948b9b746cbad21e6a286 (127.15.30.66:36761)
I20260808 04:54:38.866325 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.862388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:40865 (local address 127.15.30.65:34003)
0808 04:54:38.862515 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.862519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.862537 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:54:38.862696 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.862700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.862788 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.862878 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:38.863204 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.863784 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.865181 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.865750 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.865885 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.865936 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.866122 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.866198 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:54:38.866326 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.862162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34003 (local address 127.15.30.67:40865)
0808 04:54:38.862429 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.862450 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.862562 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.862910 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.862913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.862936 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:38.863071 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.863078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.863901 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.863904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.865044 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.865053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.865880 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.865884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.865994 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.866013 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.866103 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.866177 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":23}
I20260808 04:54:38.866329 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.862309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:51405 (local address 127.15.30.66:36761)
0808 04:54:38.862447 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.862452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.862830 (+ 378us) server_negotiation.cc:407] Connection header received
0808 04:54:38.862934 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.862937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.862992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.863079 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:38.863398 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.863917 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.865206 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.865751 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.865828 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.865867 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.866140 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.866207 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:54:38.866640 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.862252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:36761 (local address 127.15.30.67:51405)
0808 04:54:38.862708 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.862723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:38.862845 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:38.863102 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:38.863104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:38.863119 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:38.863278 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.863286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.864031 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.864034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.865060 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:38.865070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.865946 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.865948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:38.866024 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.866039 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 04:54:38.866122 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:54:38.866190 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":3,"thread_start_us":155,"threads_started":1}
I20260808 04:54:38.867030 11834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f128b012ebf248a6aed892a3fcce7249" candidate_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" is_pre_election: true
I20260808 04:54:38.867316 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f128b012ebf248a6aed892a3fcce7249" candidate_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c43b61fb5948b9b746cbad21e6a286" is_pre_election: true
I20260808 04:54:38.867342 11834 raft_consensus.cc:2454] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2480d0d939f745e8b72f8b8f76bb8da2 in term 0.
I20260808 04:54:38.867596 12051 raft_consensus.cc:2454] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2480d0d939f745e8b72f8b8f76bb8da2 in term 0.
I20260808 04:54:38.867888 12166 leader_election.cc:302] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2480d0d939f745e8b72f8b8f76bb8da2, c8f21a8246f940ebb0b42c1aadf854f2; no voters:
I20260808 04:54:38.868216 12373 raft_consensus.cc:2781] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:54:38.868299 12373 raft_consensus.cc:482] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:54:38.868330 12373 raft_consensus.cc:3037] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:38.869510 12373 raft_consensus.cc:504] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.869879 12373 leader_election.cc:288] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [CANDIDATE]: Term 1 election: Requested vote from peers c8f21a8246f940ebb0b42c1aadf854f2 (127.15.30.65:34003), 61c43b61fb5948b9b746cbad21e6a286 (127.15.30.66:36761)
I20260808 04:54:38.870253 11834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f128b012ebf248a6aed892a3fcce7249" candidate_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f21a8246f940ebb0b42c1aadf854f2"
I20260808 04:54:38.870302 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f128b012ebf248a6aed892a3fcce7249" candidate_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c43b61fb5948b9b746cbad21e6a286"
I20260808 04:54:38.870368 11834 raft_consensus.cc:3037] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:38.870450 12051 raft_consensus.cc:3037] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:38.871419 11834 raft_consensus.cc:2454] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2480d0d939f745e8b72f8b8f76bb8da2 in term 1.
I20260808 04:54:38.871623 12051 raft_consensus.cc:2454] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2480d0d939f745e8b72f8b8f76bb8da2 in term 1.
I20260808 04:54:38.871747 12166 leader_election.cc:302] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2480d0d939f745e8b72f8b8f76bb8da2, c8f21a8246f940ebb0b42c1aadf854f2; no voters:
I20260808 04:54:38.871946 12373 raft_consensus.cc:2781] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:38.872195 12373 raft_consensus.cc:686] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 1 LEADER]: Becoming Leader. State: Replica: 2480d0d939f745e8b72f8b8f76bb8da2, State: Running, Role: LEADER
I20260808 04:54:38.872567 12373 consensus_queue.cc:237] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } }
I20260808 04:54:38.875890 11588 catalog_manager.cc:5495] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2480d0d939f745e8b72f8b8f76bb8da2 (127.15.30.67). New cstate: current_term: 1 leader_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2480d0d939f745e8b72f8b8f76bb8da2" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 } health_report { overall_health: UNKNOWN } } }
W20260808 04:54:38.883404 12342 tablet.cc:2369] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:38.918995 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.915711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.15.30.67:46715)
0808 04:54:38.915854 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:38.915858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:38.915872 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:54:38.915964 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:38.915966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:38.916020 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:38.916106 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:54:38.916325 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.916878 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.917679 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:38.917894 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:38.917970 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:38.918254 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:54:38.918645 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:54:38.918754 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:38.918836 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 04:54:38.918873 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 04:54:38.922428 12133 tablet.cc:2369] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:38.924887 12051 raft_consensus.cc:1261] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 1 FOLLOWER]: Refusing update from remote peer 2480d0d939f745e8b72f8b8f76bb8da2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:38.924896 11834 raft_consensus.cc:1261] T f128b012ebf248a6aed892a3fcce7249 P c8f21a8246f940ebb0b42c1aadf854f2 [term 1 FOLLOWER]: Refusing update from remote peer 2480d0d939f745e8b72f8b8f76bb8da2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:38.925588 12373 consensus_queue.cc:1035] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c43b61fb5948b9b746cbad21e6a286" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:38.925838 12376 consensus_queue.cc:1035] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8f21a8246f940ebb0b42c1aadf854f2" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:38.932025 12380 mvcc.cc:205] Tried to move back new op lower bound from 7316131344069279744 to 7316131343864573952. Current Snapshot: MvccSnapshot[applied={T|T < 7316131344069279744}]
I20260808 04:54:38.941509 11588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:38.941646 11588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:38.943946 11588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ff0c857c79f429d9636ad8a0a727369: Sending DeleteTablet for 3 replicas of tablet f128b012ebf248a6aed892a3fcce7249
I20260808 04:54:38.944696 11806 tablet_service.cc:1514] Processing DeleteTablet for tablet f128b012ebf248a6aed892a3fcce7249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:38 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:60666
I20260808 04:54:38.944763 12229 tablet_service.cc:1514] Processing DeleteTablet for tablet f128b012ebf248a6aed892a3fcce7249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:38 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:51442
I20260808 04:54:38.944840 12020 tablet_service.cc:1514] Processing DeleteTablet for tablet f128b012ebf248a6aed892a3fcce7249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:38 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:39696
I20260808 04:54:38.945226 12396 tablet_replica.cc:330] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: stopping tablet replica
I20260808 04:54:38.945535 12396 raft_consensus.cc:2229] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:54:38.945756 12397 tablet_replica.cc:330] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: stopping tablet replica
I20260808 04:54:38.945855 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 11699
I20260808 04:54:38.945863 12396 raft_consensus.cc:2258] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:38.946048 12397 raft_consensus.cc:2229] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:38.946323 12397 raft_consensus.cc:2258] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:38.947594 12396 ts_tablet_manager.cc:1905] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:38.947999 12397 ts_tablet_manager.cc:1905] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:38.951468 12396 ts_tablet_manager.cc:1918] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:38.951560 12396 log.cc:1196] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/f128b012ebf248a6aed892a3fcce7249
I20260808 04:54:38.951902 12396 ts_tablet_manager.cc:1939] T f128b012ebf248a6aed892a3fcce7249 P 2480d0d939f745e8b72f8b8f76bb8da2: Deleting consensus metadata
I20260808 04:54:38.952960 12397 ts_tablet_manager.cc:1918] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:38.953087 12397 log.cc:1196] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/f128b012ebf248a6aed892a3fcce7249
I20260808 04:54:38.953161 11565 catalog_manager.cc:4841] TS 2480d0d939f745e8b72f8b8f76bb8da2 (127.15.30.67:46715): tablet f128b012ebf248a6aed892a3fcce7249 (table test-table [id=e23c6b7397c54d53b4a5e965806ecfb2]) successfully deleted
I20260808 04:54:38.953507 12397 ts_tablet_manager.cc:1939] T f128b012ebf248a6aed892a3fcce7249 P 61c43b61fb5948b9b746cbad21e6a286: Deleting consensus metadata
I20260808 04:54:38.955044 11565 catalog_manager.cc:4841] TS 61c43b61fb5948b9b746cbad21e6a286 (127.15.30.66:36761): tablet f128b012ebf248a6aed892a3fcce7249 (table test-table [id=e23c6b7397c54d53b4a5e965806ecfb2]) successfully deleted
W20260808 04:54:38.965143 11564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:34003 (error 108)
I20260808 04:54:38.965305 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 11912
I20260808 04:54:38.968336 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:38.967961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:34003 (local address 127.0.0.1:60672)
0808 04:54:38.968125 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:38.968207 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 04:54:38.968524 11564 catalog_manager.cc:4584] TS c8f21a8246f940ebb0b42c1aadf854f2 (127.15.30.65:34003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f128b012ebf248a6aed892a3fcce7249: Network error: Client connection negotiation failed: client connection to 127.15.30.65:34003: connect: Connection refused (error 111)
I20260808 04:54:38.975003 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 12139
I20260808 04:54:38.988694 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 11536
2026-08-08T04:54:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T04:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:39Z Disabled control of system clock
I20260808 04:54:39.022786 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:46639
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:46639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:54:39.138650 12406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:39.138908 12406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:39.138943 12406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:39.142868 12406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:39.142923 12406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:39.142938 12406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:39.142956 12406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:39.142970 12406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:39.147676 12406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:46639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:46639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:39.148736 12406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:39.150087 12406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:39.158380 12419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:39.158396 12423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:39.158409 12415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:39.158715 12406 server_base.cc:1034] running on GCE node
I20260808 04:54:39.159303 12406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:39.160352 12406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:39.161571 12406 hybrid_clock.cc:648] HybridClock initialized: now 1786164879161549 us; error 40 us; skew 500 ppm
I20260808 04:54:39.163550 12406 webserver.cc:460] Webserver started at http://127.15.30.126:37399/ using document root <none> and password file <none>
I20260808 04:54:39.164116 12406 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:39.164179 12406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:39.164477 12406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:39.166491 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "44ca2765dd3c316bad8f9b03c971c358"
server_key_iv: "fe4b851746cecb35d2acb4e6978e3ec9"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.167061 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "44ca2765dd3c316bad8f9b03c971c358"
server_key_iv: "fe4b851746cecb35d2acb4e6978e3ec9"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.170621 12406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:39.172981 12434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.174149 12406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:39.174283 12406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "44ca2765dd3c316bad8f9b03c971c358"
server_key_iv: "fe4b851746cecb35d2acb4e6978e3ec9"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.174383 12406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:39.186280 12406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:39.186913 12406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:39.187121 12406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:39.194370 12406 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:46639
I20260808 04:54:39.194373 12517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:46639 every 8 connection(s)
I20260808 04:54:39.195349 12406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:39.198401 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 12406
I20260808 04:54:39.198539 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:39.198678 12519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:39.198884 15481 external_mini_cluster.cc:1428] Setting key 6ee00d4ff7161b4187a5b129e35be972
I20260808 04:54:39.205024 12519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa: Bootstrap starting.
I20260808 04:54:39.207170 12519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:39.208029 12519 log.cc:824] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:39.208237 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.200285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.15.30.126:46639)
0808 04:54:39.200947 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.200962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.200996 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:54:39.201927 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.201948 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.202329 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.202658 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:39.203082 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.204009 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.205007 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.205523 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.205704 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.206583 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:39.206589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:39.206600 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:39.206972 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:39.207024 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:39.207043 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:39.207150 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.207268 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.207517 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":149,"threads_started":1}
I20260808 04:54:39.210199 12519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa: No bootstrap required, opened a new log
I20260808 04:54:39.213649 12519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46639 } }
I20260808 04:54:39.213912 12519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:39.213995 12519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7ffe6c0bcf41ccaba2bc7930f533aa, State: Initialized, Role: FOLLOWER
I20260808 04:54:39.214398 12519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46639 } }
I20260808 04:54:39.214551 12519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:39.214649 12519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:39.214785 12519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:39.215643 12519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46639 } }
I20260808 04:54:39.215993 12519 leader_election.cc:302] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a7ffe6c0bcf41ccaba2bc7930f533aa; no voters:
I20260808 04:54:39.216301 12519 leader_election.cc:288] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:39.216427 12525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:39.216741 12525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [term 1 LEADER]: Becoming Leader. State: Replica: 4a7ffe6c0bcf41ccaba2bc7930f533aa, State: Running, Role: LEADER
I20260808 04:54:39.217056 12525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46639 } }
I20260808 04:54:39.217562 12519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:39.218905 12526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46639 } } }
I20260808 04:54:39.218892 12527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a7ffe6c0bcf41ccaba2bc7930f533aa. Latest consensus state: current_term: 1 leader_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7ffe6c0bcf41ccaba2bc7930f533aa" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 46639 } } }
I20260808 04:54:39.219043 12526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:39.219043 12527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:39.219650 12538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:39.222545 12538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:39.227407 12538 catalog_manager.cc:1347] Generated new cluster ID: 485d42659f9549858104b17daba56aa0
I20260808 04:54:39.227468 12538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:39.235924 12538 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:39.237073 12538 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:39.249845 12538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa: Generated new TSK 0
I20260808 04:54:39.250578 12538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:39.256289 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46639
--builtin_ntp_servers=127.15.30.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:54:39.366251 12552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:39.366513 12552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:39.366590 12552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:39.370715 12552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:39.370808 12552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:39.370896 12552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:54:39.375813 12552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:39.376816 12552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:39.378190 12552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:39.387552 12564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:39.387574 12566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:39.387571 12569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:39.387979 12552 server_base.cc:1034] running on GCE node
I20260808 04:54:39.388334 12552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:39.388967 12552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:39.390127 12552 hybrid_clock.cc:648] HybridClock initialized: now 1786164879390113 us; error 27 us; skew 500 ppm
I20260808 04:54:39.391968 12552 webserver.cc:460] Webserver started at http://127.15.30.65:37273/ using document root <none> and password file <none>
I20260808 04:54:39.392581 12552 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:39.392652 12552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:39.392834 12552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:39.394771 12552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "8b28aa7d46de4ad58b7b808542a96a06"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "2ec849b0034ae09884b5129175a9d7e9"
server_key_iv: "eb8ddb6acc2ddd628a836323e7af66a0"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.395283 12552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "8b28aa7d46de4ad58b7b808542a96a06"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "2ec849b0034ae09884b5129175a9d7e9"
server_key_iv: "eb8ddb6acc2ddd628a836323e7af66a0"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.398851 12552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:54:39.401198 12578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.402464 12552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:39.402603 12552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "8b28aa7d46de4ad58b7b808542a96a06"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "2ec849b0034ae09884b5129175a9d7e9"
server_key_iv: "eb8ddb6acc2ddd628a836323e7af66a0"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.402725 12552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:39.423774 12552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:39.424494 12552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:39.424697 12552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:39.425282 12552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:39.426216 12552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:39.426298 12552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.426368 12552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:39.426419 12552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.434924 12552 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:44799
I20260808 04:54:39.434953 12755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:44799 every 8 connection(s)
I20260808 04:54:39.435767 12552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:54:39.443459 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 12552
I20260808 04:54:39.443602 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:54:39.443986 15481 external_mini_cluster.cc:1428] Setting key 04e2639a2960cab2ae9f38bb5f83fdc3
I20260808 04:54:39.445237 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.437614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:57757 (local address 127.15.30.126:46639)
0808 04:54:39.437789 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.437794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.439066 (+ 1272us) server_negotiation.cc:407] Connection header received
0808 04:54:39.440477 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.440479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.440526 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.440602 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:39.441447 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.441969 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.443044 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.443275 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.443357 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.444688 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:39.444692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:39.444695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:39.444710 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:39.444734 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:39.444743 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:39.444801 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.445001 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.445059 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:54:39.446192 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.438057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:46639 (local address 127.15.30.65:57757)
0808 04:54:39.438851 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:39.438944 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:39.440269 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:39.440765 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:39.440781 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:39.440863 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:39.441327 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.441339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.442077 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.442079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.442935 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.442944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.443449 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.443454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.443675 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.444353 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:39.444355 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:39.444436 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:39.444524 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:39.444831 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:39.444844 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:39.444851 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:54:39.444973 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 04:54:39.445183 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":116,"threads_started":1}
I20260808 04:54:39.446314 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46639
--builtin_ntp_servers=127.15.30.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:54:39.447499 12757 heartbeater.cc:347] Connected to a master server at 127.15.30.126:46639
I20260808 04:54:39.447751 12757 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:39.448350 12757 heartbeater.cc:511] Master 127.15.30.126:46639 requested a full tablet report, sending...
I20260808 04:54:39.450034 12460 ts_manager.cc:194] Registered new tserver with Master: 8b28aa7d46de4ad58b7b808542a96a06 (127.15.30.65:44799)
I20260808 04:54:39.451097 12460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.65:57757
W20260808 04:54:39.558782 12761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:39.559010 12761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:39.559046 12761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:39.564247 12761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:39.564323 12761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:39.564433 12761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:54:39.569208 12761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:39.570160 12761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:39.571458 12761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:39.581127 12768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:39.581142 12771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:39.581127 12773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:39.581391 12761 server_base.cc:1034] running on GCE node
I20260808 04:54:39.581924 12761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:39.582557 12761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:39.583770 12761 hybrid_clock.cc:648] HybridClock initialized: now 1786164879583752 us; error 25 us; skew 500 ppm
I20260808 04:54:39.585718 12761 webserver.cc:460] Webserver started at http://127.15.30.66:46721/ using document root <none> and password file <none>
I20260808 04:54:39.586299 12761 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:39.586360 12761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:39.586612 12761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:39.588727 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "49d11cba4fe44b1a9ad4d07e43b89210"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "f0fb3158d0652ba471c7af7512ec6c07"
server_key_iv: "0b25cd2fdaa021471b65b93f109b9ca5"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.589339 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "49d11cba4fe44b1a9ad4d07e43b89210"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "f0fb3158d0652ba471c7af7512ec6c07"
server_key_iv: "0b25cd2fdaa021471b65b93f109b9ca5"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.592928 12761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:54:39.595320 12786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.596499 12761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:39.596616 12761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "49d11cba4fe44b1a9ad4d07e43b89210"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "f0fb3158d0652ba471c7af7512ec6c07"
server_key_iv: "0b25cd2fdaa021471b65b93f109b9ca5"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.596711 12761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:39.623745 12761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:39.624491 12761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:39.624698 12761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:39.625336 12761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:39.626271 12761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:39.626317 12761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.626384 12761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:39.626418 12761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.635030 12761 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:43723
I20260808 04:54:39.635079 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:43723 every 8 connection(s)
I20260808 04:54:39.635887 12761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:54:39.644985 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.637730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:51755 (local address 127.15.30.126:46639)
0808 04:54:39.637882 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.637886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.638908 (+ 1022us) server_negotiation.cc:407] Connection header received
0808 04:54:39.640296 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.640299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.640347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.640500 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:39.641375 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.641907 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.643035 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.643309 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.643401 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.644496 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:39.644499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:39.644501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:39.644513 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:39.644532 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:39.644540 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:39.644645 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.644810 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.644865 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:54:39.645763 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.638122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:46639 (local address 127.15.30.66:51755)
0808 04:54:39.638746 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:39.638797 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:39.640061 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:39.640663 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:39.640669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:39.640752 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:39.641221 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.641234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.642023 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.642026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.642897 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.642905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.643453 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.643456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.643597 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.644189 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:39.644190 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:39.644228 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:39.644300 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:39.644666 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:39.644674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:39.644679 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:54:39.644795 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:54:39.645000 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260808 04:54:39.646126 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 12761
I20260808 04:54:39.646286 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:54:39.646595 15481 external_mini_cluster.cc:1428] Setting key dad11b72fa4f018e5bed855f38c6462d
I20260808 04:54:39.647181 12962 heartbeater.cc:347] Connected to a master server at 127.15.30.126:46639
I20260808 04:54:39.647449 12962 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:39.648051 12962 heartbeater.cc:511] Master 127.15.30.126:46639 requested a full tablet report, sending...
I20260808 04:54:39.648792 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46639
--builtin_ntp_servers=127.15.30.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:54:39.649302 12460 ts_manager.cc:194] Registered new tserver with Master: 49d11cba4fe44b1a9ad4d07e43b89210 (127.15.30.66:43723)
I20260808 04:54:39.649906 12460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.66:51755
W20260808 04:54:39.763808 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:39.764127 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:39.764170 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:39.768172 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:39.768222 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:39.768273 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:54:39.772861 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:46639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:39.773814 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:39.775146 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:39.784716 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:39.784746 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:39.784795 12967 server_base.cc:1034] running on GCE node
W20260808 04:54:39.784746 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:39.785413 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:39.786022 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:39.787241 12967 hybrid_clock.cc:648] HybridClock initialized: now 1786164879787208 us; error 34 us; skew 500 ppm
I20260808 04:54:39.789127 12967 webserver.cc:460] Webserver started at http://127.15.30.67:33445/ using document root <none> and password file <none>
I20260808 04:54:39.789708 12967 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:39.789806 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:39.790032 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:39.792032 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "e7415d575c40487aaf2a5f92880fd7a1"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "7554d8d8d0407623fb2e5a68d15f07d6"
server_key_iv: "6fe4039b1abe1876e989acb6d6697589"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.792649 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "e7415d575c40487aaf2a5f92880fd7a1"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "7554d8d8d0407623fb2e5a68d15f07d6"
server_key_iv: "6fe4039b1abe1876e989acb6d6697589"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.796271 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:54:39.798590 12992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.799772 12967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:39.799866 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "e7415d575c40487aaf2a5f92880fd7a1"
format_stamp: "Formatted at 2026-08-08 04:54:39 on dist-test-slave-zpvx"
server_key: "7554d8d8d0407623fb2e5a68d15f07d6"
server_key_iv: "6fe4039b1abe1876e989acb6d6697589"
server_key_version: "encryptionkey@0"
I20260808 04:54:39.799947 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:39.815400 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:39.816005 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:39.816148 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:39.816713 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:39.817557 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:39.817616 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.817654 12967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:39.817703 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:39.826325 12967 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:32769
I20260808 04:54:39.826401 13179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:32769 every 8 connection(s)
I20260808 04:54:39.827211 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:54:39.836161 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.829065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:34109 (local address 127.15.30.126:46639)
0808 04:54:39.829208 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.829211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.830224 (+ 1013us) server_negotiation.cc:407] Connection header received
0808 04:54:39.831556 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.831559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.831611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.831687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:39.832618 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.833138 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.834253 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.834467 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.834556 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.835738 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:39.835741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:39.835743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:39.835754 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:39.835773 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:39.835780 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:39.835843 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.835991 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.836039 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:54:39.836515 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 12967
I20260808 04:54:39.836666 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:54:39.836959 15481 external_mini_cluster.cc:1428] Setting key 5f7ef2f2fa6a5c09d1047042fb752dfc
I20260808 04:54:39.837172 13184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.829448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:46639 (local address 127.15.30.67:34109)
0808 04:54:39.830075 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:39.830113 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:39.831348 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:39.831863 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:39.831870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:39.831950 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:39.832473 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.832489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.833250 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.833252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.834109 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.834128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.834607 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.834609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.834744 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.835475 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:39.835477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:39.835518 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:39.835579 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:39.835862 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:39.835869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:39.835874 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:54:39.835968 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:54:39.836216 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260808 04:54:39.838310 13180 heartbeater.cc:347] Connected to a master server at 127.15.30.126:46639
I20260808 04:54:39.838533 13180 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:39.839054 13180 heartbeater.cc:511] Master 127.15.30.126:46639 requested a full tablet report, sending...
I20260808 04:54:39.840081 12457 ts_manager.cc:194] Registered new tserver with Master: e7415d575c40487aaf2a5f92880fd7a1 (127.15.30.67:32769)
I20260808 04:54:39.840708 12457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.67:34109
I20260808 04:54:39.851154 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:54:39.857146 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.853188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.15.30.126:46639)
0808 04:54:39.853329 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.853336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.853454 (+ 118us) server_negotiation.cc:407] Connection header received
0808 04:54:39.853544 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.853546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.853597 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.853662 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:39.853914 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.854396 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.855192 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.855391 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.855470 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.856589 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:39.856592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:39.856594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:39.856605 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:39.856621 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:39.856628 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:39.856738 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.856915 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.856982 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:54:39.860123 12457 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:54:39.863072 12457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:54:39.877552 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.872437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.15.30.65:44799)
0808 04:54:39.872841 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.872849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.872869 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:54:39.872929 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.872932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.873069 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.873179 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:39.873438 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.874535 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.876232 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.876859 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.876982 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.877171 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.877312 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.877404 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":123,"threads_started":1}
I20260808 04:54:39.877614 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.871361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:43723 (local address 127.0.0.1:40144)
0808 04:54:39.872200 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:39.872217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:39.872396 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:39.872934 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:39.872940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:39.872964 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:39.873113 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.873120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.874672 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.874675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.876055 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.876065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.877184 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.877188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.877287 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.877312 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:54:39.877370 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 04:54:39.877438 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":19,"thread_start_us":97,"threads_started":1}
I20260808 04:54:39.877514 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.871975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:44799 (local address 127.0.0.1:59046)
0808 04:54:39.872615 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:39.872627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:39.872701 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:39.873192 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:39.873197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:39.873220 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:39.873319 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.873325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.874672 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.874675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.876095 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.876106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.876983 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.876987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.877088 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.877124 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:54:39.877240 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:54:39.877365 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260808 04:54:39.878394 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.871810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.15.30.67:32769)
0808 04:54:39.873461 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.873470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.873490 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:54:39.873551 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.873555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.873689 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.873846 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:39.874077 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.875001 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.876314 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.877036 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.877157 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.877347 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.878113 (+ 766us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.878208 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1531,"thread_start_us":98,"threads_started":1}
I20260808 04:54:39.878443 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.871968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.15.30.66:43723)
0808 04:54:39.872402 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:39.872414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:39.872445 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:54:39.872527 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:39.872534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:39.872752 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:39.872952 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:39.873278 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.874540 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.876199 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.877035 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.877600 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.877817 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:39.877868 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 04:54:39.878283 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":105,"threads_started":1}
I20260808 04:54:39.878679 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:39.871270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:32769 (local address 127.0.0.1:42598)
0808 04:54:39.871600 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:39.871643 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:39.872559 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:39.873822 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:39.873825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:39.873843 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:39.873955 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.873962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.875111 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.875114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.876219 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:39.876228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:39.877410 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:39.877414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:39.877503 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:39.877526 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:54:39.877737 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 04:54:39.877809 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260808 04:54:39.880978 12861 tablet_service.cc:1467] Processing CreateTablet for tablet 75781cc0ff5242409ea02d9bb9cb0edc (DEFAULT_TABLE table=test-table [id=a29d8c7320ac4ce693318f4d64cb7964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:39.881021 12652 tablet_service.cc:1467] Processing CreateTablet for tablet 75781cc0ff5242409ea02d9bb9cb0edc (DEFAULT_TABLE table=test-table [id=a29d8c7320ac4ce693318f4d64cb7964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:39.881181 13071 tablet_service.cc:1467] Processing CreateTablet for tablet 75781cc0ff5242409ea02d9bb9cb0edc (DEFAULT_TABLE table=test-table [id=a29d8c7320ac4ce693318f4d64cb7964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:39.882074 12652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75781cc0ff5242409ea02d9bb9cb0edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:39.882074 13071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75781cc0ff5242409ea02d9bb9cb0edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:39.882076 12861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75781cc0ff5242409ea02d9bb9cb0edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:39.888192 13210 tablet_bootstrap.cc:492] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Bootstrap starting.
I20260808 04:54:39.889569 13216 tablet_bootstrap.cc:492] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06: Bootstrap starting.
I20260808 04:54:39.890619 13210 tablet_bootstrap.cc:654] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:39.891291 13216 tablet_bootstrap.cc:654] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:39.891384 13210 log.cc:824] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:39.892020 13211 tablet_bootstrap.cc:492] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Bootstrap starting.
I20260808 04:54:39.892099 13216 log.cc:824] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:39.894054 13216 tablet_bootstrap.cc:492] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06: No bootstrap required, opened a new log
I20260808 04:54:39.894068 13210 tablet_bootstrap.cc:492] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: No bootstrap required, opened a new log
I20260808 04:54:39.894256 13216 ts_tablet_manager.cc:1397] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260808 04:54:39.894263 13210 ts_tablet_manager.cc:1397] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:54:39.894560 13211 tablet_bootstrap.cc:654] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:39.895223 13211 log.cc:824] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:39.896891 13211 tablet_bootstrap.cc:492] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: No bootstrap required, opened a new log
I20260808 04:54:39.897089 13211 ts_tablet_manager.cc:1397] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260808 04:54:39.898221 13210 raft_consensus.cc:348] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:39.898295 13216 raft_consensus.cc:348] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:39.898501 13210 raft_consensus.cc:374] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:39.898500 13216 raft_consensus.cc:374] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:39.898559 13216 raft_consensus.cc:729] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b28aa7d46de4ad58b7b808542a96a06, State: Initialized, Role: FOLLOWER
I20260808 04:54:39.898567 13210 raft_consensus.cc:729] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d11cba4fe44b1a9ad4d07e43b89210, State: Initialized, Role: FOLLOWER
I20260808 04:54:39.899020 13210 consensus_queue.cc:260] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:39.899020 13216 consensus_queue.cc:260] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:39.899997 13216 ts_tablet_manager.cc:1428] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260808 04:54:39.899997 13210 ts_tablet_manager.cc:1428] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:54:39.900410 12757 heartbeater.cc:503] Master 127.15.30.126:46639 was elected leader, sending a full tablet report...
I20260808 04:54:39.901276 13211 raft_consensus.cc:348] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:39.901513 13211 raft_consensus.cc:374] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:39.901506 12962 heartbeater.cc:503] Master 127.15.30.126:46639 was elected leader, sending a full tablet report...
I20260808 04:54:39.901563 13211 raft_consensus.cc:729] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7415d575c40487aaf2a5f92880fd7a1, State: Initialized, Role: FOLLOWER
I20260808 04:54:39.901984 13211 consensus_queue.cc:260] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:39.902957 13180 heartbeater.cc:503] Master 127.15.30.126:46639 was elected leader, sending a full tablet report...
I20260808 04:54:39.902969 13211 ts_tablet_manager.cc:1428] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260808 04:54:39.936996 12759 tablet.cc:2369] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:40.022006 13224 raft_consensus.cc:482] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:40.022257 13224 raft_consensus.cc:504] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:40.023432 13224 leader_election.cc:288] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d11cba4fe44b1a9ad4d07e43b89210 (127.15.30.66:43723), 8b28aa7d46de4ad58b7b808542a96a06 (127.15.30.65:44799)
I20260808 04:54:40.028520 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:40.023843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:60631 (local address 127.15.30.65:44799)
0808 04:54:40.023991 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:40.023997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:40.024225 (+ 228us) server_negotiation.cc:407] Connection header received
0808 04:54:40.024329 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:40.024332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:40.024410 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:40.024482 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:40.024794 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.025495 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.027236 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.027848 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.027977 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:40.028027 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:40.028295 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 04:54:40.028384 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:54:40.028520 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:40.023772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:37663 (local address 127.15.30.66:43723)
0808 04:54:40.023899 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:40.023904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:40.023920 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:54:40.024114 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:40.024117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:40.024168 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:40.024285 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:40.024656 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.025467 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.027236 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.027836 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.027977 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:40.028027 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:40.028295 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 04:54:40.028387 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:54:40.028546 13184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:40.023640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:43723 (local address 127.15.30.67:37663)
0808 04:54:40.023845 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:40.023862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:40.023956 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:40.024313 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:40.024319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:40.024340 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:40.024516 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:40.024524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.025602 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.025605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:40.027018 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:40.027023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.027982 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.027986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:40.028141 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:40.028182 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 04:54:40.028264 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:54:40.028389 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":166}
I20260808 04:54:40.028554 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:40.023684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:44799 (local address 127.15.30.67:60631)
0808 04:54:40.024129 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:40.024139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:40.024221 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:40.024516 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:40.024519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:40.024535 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:40.024671 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:40.024677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.025602 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.025605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:40.027011 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:40.027023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.027985 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.027988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:40.028137 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:40.028176 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:54:40.028258 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:54:40.028391 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":175,"thread_start_us":156,"threads_started":1}
I20260808 04:54:40.029315 12898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75781cc0ff5242409ea02d9bb9cb0edc" candidate_uuid: "e7415d575c40487aaf2a5f92880fd7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" is_pre_election: true
I20260808 04:54:40.029376 12686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75781cc0ff5242409ea02d9bb9cb0edc" candidate_uuid: "e7415d575c40487aaf2a5f92880fd7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b28aa7d46de4ad58b7b808542a96a06" is_pre_election: true
I20260808 04:54:40.029605 12898 raft_consensus.cc:2454] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7415d575c40487aaf2a5f92880fd7a1 in term 0.
I20260808 04:54:40.029650 12686 raft_consensus.cc:2454] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7415d575c40487aaf2a5f92880fd7a1 in term 0.
I20260808 04:54:40.030086 12995 leader_election.cc:302] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d11cba4fe44b1a9ad4d07e43b89210, e7415d575c40487aaf2a5f92880fd7a1; no voters:
I20260808 04:54:40.030370 13224 raft_consensus.cc:2781] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:54:40.030463 13224 raft_consensus.cc:482] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:54:40.030496 13224 raft_consensus.cc:3037] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:40.031544 13224 raft_consensus.cc:504] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:40.031906 13224 leader_election.cc:288] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [CANDIDATE]: Term 1 election: Requested vote from peers 49d11cba4fe44b1a9ad4d07e43b89210 (127.15.30.66:43723), 8b28aa7d46de4ad58b7b808542a96a06 (127.15.30.65:44799)
I20260808 04:54:40.032220 12686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75781cc0ff5242409ea02d9bb9cb0edc" candidate_uuid: "e7415d575c40487aaf2a5f92880fd7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b28aa7d46de4ad58b7b808542a96a06"
I20260808 04:54:40.032220 12898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75781cc0ff5242409ea02d9bb9cb0edc" candidate_uuid: "e7415d575c40487aaf2a5f92880fd7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d11cba4fe44b1a9ad4d07e43b89210"
I20260808 04:54:40.032351 12686 raft_consensus.cc:3037] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:40.032347 12898 raft_consensus.cc:3037] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:40.033649 12686 raft_consensus.cc:2454] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7415d575c40487aaf2a5f92880fd7a1 in term 1.
I20260808 04:54:40.033649 12898 raft_consensus.cc:2454] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7415d575c40487aaf2a5f92880fd7a1 in term 1.
I20260808 04:54:40.033993 12995 leader_election.cc:302] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d11cba4fe44b1a9ad4d07e43b89210, e7415d575c40487aaf2a5f92880fd7a1; no voters:
I20260808 04:54:40.034189 13224 raft_consensus.cc:2781] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:40.034401 13224 raft_consensus.cc:686] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 1 LEADER]: Becoming Leader. State: Replica: e7415d575c40487aaf2a5f92880fd7a1, State: Running, Role: LEADER
I20260808 04:54:40.034716 13224 consensus_queue.cc:237] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } }
I20260808 04:54:40.037861 12457 catalog_manager.cc:5495] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7415d575c40487aaf2a5f92880fd7a1 (127.15.30.67). New cstate: current_term: 1 leader_uuid: "e7415d575c40487aaf2a5f92880fd7a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7415d575c40487aaf2a5f92880fd7a1" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 32769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 } health_report { overall_health: UNKNOWN } } }
I20260808 04:54:40.063573 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:40.060444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.15.30.67:32769)
0808 04:54:40.060567 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:40.060571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:40.060584 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:54:40.060680 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:40.060682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:40.060732 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:40.060810 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:54:40.061034 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.061517 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.062310 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:40.062504 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:40.062578 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:40.062870 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:54:40.063226 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:54:40.063340 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:40.063429 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 04:54:40.063458 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:54:40.069171 12686 raft_consensus.cc:1261] T 75781cc0ff5242409ea02d9bb9cb0edc P 8b28aa7d46de4ad58b7b808542a96a06 [term 1 FOLLOWER]: Refusing update from remote peer e7415d575c40487aaf2a5f92880fd7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:40.069170 12898 raft_consensus.cc:1261] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 1 FOLLOWER]: Refusing update from remote peer e7415d575c40487aaf2a5f92880fd7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:40.069844 13224 consensus_queue.cc:1035] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d11cba4fe44b1a9ad4d07e43b89210" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:40.070053 13227 consensus_queue.cc:1035] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b28aa7d46de4ad58b7b808542a96a06" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:40.078684 13230 mvcc.cc:205] Tried to move back new op lower bound from 7316131348756176896 to 7316131348624723968. Current Snapshot: MvccSnapshot[applied={T|T < 7316131348756176896}]
I20260808 04:54:40.079537 13239 mvcc.cc:205] Tried to move back new op lower bound from 7316131348756176896 to 7316131348624723968. Current Snapshot: MvccSnapshot[applied={T|T < 7316131348756176896}]
I20260808 04:54:40.086374 12460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:40.086539 12460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:40.088917 12460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a7ffe6c0bcf41ccaba2bc7930f533aa: Sending DeleteTablet for 3 replicas of tablet 75781cc0ff5242409ea02d9bb9cb0edc
I20260808 04:54:40.089723 12861 tablet_service.cc:1514] Processing DeleteTablet for tablet 75781cc0ff5242409ea02d9bb9cb0edc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:40 UTC) from {username='slave'} at 127.0.0.1:40144
I20260808 04:54:40.089767 12652 tablet_service.cc:1514] Processing DeleteTablet for tablet 75781cc0ff5242409ea02d9bb9cb0edc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:40 UTC) from {username='slave'} at 127.0.0.1:59046
I20260808 04:54:40.090245 13071 tablet_service.cc:1514] Processing DeleteTablet for tablet 75781cc0ff5242409ea02d9bb9cb0edc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:40 UTC) from {username='slave'} at 127.0.0.1:42598
I20260808 04:54:40.090433 13250 tablet_replica.cc:330] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: stopping tablet replica
I20260808 04:54:40.090682 13252 tablet_replica.cc:330] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: stopping tablet replica
I20260808 04:54:40.090801 13250 raft_consensus.cc:2229] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:40.090866 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 12552
I20260808 04:54:40.091142 13250 raft_consensus.cc:2258] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:40.091639 13252 raft_consensus.cc:2229] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:54:40.092020 13252 raft_consensus.cc:2258] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:40.097131 13252 ts_tablet_manager.cc:1905] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:40.100180 13250 ts_tablet_manager.cc:1905] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:40.100864 13252 ts_tablet_manager.cc:1918] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:40.100971 13252 log.cc:1196] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/75781cc0ff5242409ea02d9bb9cb0edc
I20260808 04:54:40.101315 13252 ts_tablet_manager.cc:1939] T 75781cc0ff5242409ea02d9bb9cb0edc P e7415d575c40487aaf2a5f92880fd7a1: Deleting consensus metadata
I20260808 04:54:40.102521 12435 catalog_manager.cc:4841] TS e7415d575c40487aaf2a5f92880fd7a1 (127.15.30.67:32769): tablet 75781cc0ff5242409ea02d9bb9cb0edc (table test-table [id=a29d8c7320ac4ce693318f4d64cb7964]) successfully deleted
I20260808 04:54:40.103132 13250 ts_tablet_manager.cc:1918] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:40.103232 13250 log.cc:1196] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/75781cc0ff5242409ea02d9bb9cb0edc
I20260808 04:54:40.103550 13250 ts_tablet_manager.cc:1939] T 75781cc0ff5242409ea02d9bb9cb0edc P 49d11cba4fe44b1a9ad4d07e43b89210: Deleting consensus metadata
I20260808 04:54:40.104725 12439 catalog_manager.cc:4841] TS 49d11cba4fe44b1a9ad4d07e43b89210 (127.15.30.66:43723): tablet 75781cc0ff5242409ea02d9bb9cb0edc (table test-table [id=a29d8c7320ac4ce693318f4d64cb7964]) successfully deleted
W20260808 04:54:40.106591 12435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:44799 (error 108)
I20260808 04:54:40.107470 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 12761
I20260808 04:54:40.107661 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:40.107350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:44799 (local address 127.0.0.1:59052)
0808 04:54:40.107475 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:40.107557 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:44799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 04:54:40.118494 12435 catalog_manager.cc:4584] TS 8b28aa7d46de4ad58b7b808542a96a06 (127.15.30.65:44799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75781cc0ff5242409ea02d9bb9cb0edc: Network error: Client connection negotiation failed: client connection to 127.15.30.65:44799: connect: Connection refused (error 111)
I20260808 04:54:40.128197 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 12967
I20260808 04:54:40.140763 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 12406
2026-08-08T04:54:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1151 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:40 dist-test-slave-zpvx krb5kdc[13257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:40 dist-test-slave-zpvx krb5kdc[13257](info): set up 2 sockets
Aug 08 04:54:40 dist-test-slave-zpvx krb5kdc[13257](info): commencing operation
krb5kdc: starting...
W20260808 04:54:41.385941 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:41 dist-test-slave-zpvx krb5kdc[13257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:54:41.511437 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35651
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:35651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:41.622035 13273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:41.622309 13273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:41.622350 13273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:41.626348 13273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:41.626398 13273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:41.626417 13273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:41.626434 13273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:41.626451 13273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:41.631637 13273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:35651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:41.632694 13273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:41.633701 13273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:41.642318 13283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:41.642282 13279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:41.642274 13278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:41.642521 13273 server_base.cc:1034] running on GCE node
I20260808 04:54:41.643162 13273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:41.644207 13273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:41.645392 13273 hybrid_clock.cc:648] HybridClock initialized: now 1786164881645364 us; error 49 us; skew 500 ppm
Aug 08 04:54:41 dist-test-slave-zpvx krb5kdc[13257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164881, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:41.648077 13273 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:54:41.648986 13273 webserver.cc:460] Webserver started at http://127.15.30.126:40931/ using document root <none> and password file <none>
I20260808 04:54:41.649556 13273 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:41.649653 13273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:41.649971 13273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:41.651947 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "002c34fab3bb44cb8d98eeed3396163b"
format_stamp: "Formatted at 2026-08-08 04:54:41 on dist-test-slave-zpvx"
server_key: "48aeb16518601c0bdbf2f1b5f709c98d"
server_key_iv: "a1917207a0cab1989c70a2d8ffb5fc53"
server_key_version: "encryptionkey@0"
I20260808 04:54:41.652534 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "002c34fab3bb44cb8d98eeed3396163b"
format_stamp: "Formatted at 2026-08-08 04:54:41 on dist-test-slave-zpvx"
server_key: "48aeb16518601c0bdbf2f1b5f709c98d"
server_key_iv: "a1917207a0cab1989c70a2d8ffb5fc53"
server_key_version: "encryptionkey@0"
I20260808 04:54:41.656008 13273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:54:41.658413 13304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:41.659593 13273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:41.659693 13273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "002c34fab3bb44cb8d98eeed3396163b"
format_stamp: "Formatted at 2026-08-08 04:54:41 on dist-test-slave-zpvx"
server_key: "48aeb16518601c0bdbf2f1b5f709c98d"
server_key_iv: "a1917207a0cab1989c70a2d8ffb5fc53"
server_key_version: "encryptionkey@0"
I20260808 04:54:41.659785 13273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:41.673702 13273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:41.676914 13273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:41.677062 13273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:41.684147 13273 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:35651
I20260808 04:54:41.684176 13397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:35651 every 8 connection(s)
I20260808 04:54:41.685170 13273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:41.687484 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 13273
I20260808 04:54:41.687620 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:41.687888 15481 external_mini_cluster.cc:1428] Setting key 62849b4f324a3621f1d8db9fdd23e3a7
I20260808 04:54:41.688006 13399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:41.695389 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b: Bootstrap starting.
Aug 08 04:54:41 dist-test-slave-zpvx krb5kdc[13257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:41.697897 13399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:41.698850 13399 log.cc:824] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:41.701144 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b: No bootstrap required, opened a new log
I20260808 04:54:41.703318 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:41.689358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.15.30.126:35651)
0808 04:54:41.689786 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:41.689806 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:41.689847 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:54:41.690813 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:41.690838 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:41.691257 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:41.691630 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:41.692270 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:41.693749 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:41.694618 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:41.695287 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:41.695473 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:41.698209 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:41.698216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:41.698240 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:41.698271 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:41.700148 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:41.700619 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:41.700625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:41.700631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:41.700780 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:41.701127 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:41.701130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:41.701132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:41.701666 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:41.701866 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:41.702079 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 04:54:41.702365 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260808 04:54:41.705514 13399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002c34fab3bb44cb8d98eeed3396163b" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35651 } }
I20260808 04:54:41.705786 13399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:41.705845 13399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002c34fab3bb44cb8d98eeed3396163b, State: Initialized, Role: FOLLOWER
I20260808 04:54:41.706254 13399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002c34fab3bb44cb8d98eeed3396163b" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35651 } }
I20260808 04:54:41.706383 13399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:41.706430 13399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:41.706502 13399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:41.707329 13399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002c34fab3bb44cb8d98eeed3396163b" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35651 } }
I20260808 04:54:41.707613 13399 leader_election.cc:302] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002c34fab3bb44cb8d98eeed3396163b; no voters:
I20260808 04:54:41.707851 13399 leader_election.cc:288] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:41.707983 13405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:41.708267 13405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [term 1 LEADER]: Becoming Leader. State: Replica: 002c34fab3bb44cb8d98eeed3396163b, State: Running, Role: LEADER
I20260808 04:54:41.708598 13405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002c34fab3bb44cb8d98eeed3396163b" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35651 } }
I20260808 04:54:41.709095 13399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:41.710320 13406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "002c34fab3bb44cb8d98eeed3396163b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002c34fab3bb44cb8d98eeed3396163b" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35651 } } }
I20260808 04:54:41.710325 13409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 002c34fab3bb44cb8d98eeed3396163b. Latest consensus state: current_term: 1 leader_uuid: "002c34fab3bb44cb8d98eeed3396163b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002c34fab3bb44cb8d98eeed3396163b" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35651 } } }
I20260808 04:54:41.710505 13409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:41.710800 13406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:41.710887 13418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:41.713884 13418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:41.724624 13418 catalog_manager.cc:1347] Generated new cluster ID: 4023ccff815e4d8b96d3eed9fd0a960c
I20260808 04:54:41.724699 13418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:41.736733 13418 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:41.737604 13418 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:41.749914 13418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b: Generated new TSK 0
I20260808 04:54:41.750667 13418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.30.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
Entry for principal kudu/127.15.30.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab.
I20260808 04:54:41.787034 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35651
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:41.896216 13437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:41.896467 13437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:41.896509 13437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:41.900460 13437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:41.900507 13437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:41.900563 13437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:54:41.905598 13437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.65.keytab
--principal=kudu/127.15.30.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:41.906523 13437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:41.907483 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:41.917029 13449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:41.917057 13447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:41.917128 13437 server_base.cc:1034] running on GCE node
W20260808 04:54:41.917042 13446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:41.917680 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:41.918280 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:41.919445 13437 hybrid_clock.cc:648] HybridClock initialized: now 1786164881919427 us; error 29 us; skew 500 ppm
Aug 08 04:54:41 dist-test-slave-zpvx krb5kdc[13257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164881, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:41.922122 13437 init.cc:377] Logged in from keytab as kudu/127.15.30.65@KRBTEST.COM (short username kudu)
I20260808 04:54:41.922955 13437 webserver.cc:460] Webserver started at http://127.15.30.65:33123/ using document root <none> and password file <none>
I20260808 04:54:41.923518 13437 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:41.923614 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:41.923857 13437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:41.925882 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "d426339587eb42c9ab2ba530dea569d7"
format_stamp: "Formatted at 2026-08-08 04:54:41 on dist-test-slave-zpvx"
server_key: "328127625f060b892fd1b1dff1f6171a"
server_key_iv: "93f913f7990742aa61a3bebc20ab368b"
server_key_version: "encryptionkey@0"
I20260808 04:54:41.926440 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "d426339587eb42c9ab2ba530dea569d7"
format_stamp: "Formatted at 2026-08-08 04:54:41 on dist-test-slave-zpvx"
server_key: "328127625f060b892fd1b1dff1f6171a"
server_key_iv: "93f913f7990742aa61a3bebc20ab368b"
server_key_version: "encryptionkey@0"
I20260808 04:54:41.930117 13437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260808 04:54:41.932407 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:41.933566 13437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 04:54:41.933714 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "d426339587eb42c9ab2ba530dea569d7"
format_stamp: "Formatted at 2026-08-08 04:54:41 on dist-test-slave-zpvx"
server_key: "328127625f060b892fd1b1dff1f6171a"
server_key_iv: "93f913f7990742aa61a3bebc20ab368b"
server_key_version: "encryptionkey@0"
I20260808 04:54:41.933837 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:41.985644 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:41.988494 13437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:41.988699 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:41.989310 13437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:41.990139 13437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:41.990214 13437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:41.990285 13437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:41.990332 13437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:41.998996 13437 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:41495
I20260808 04:54:41.999023 13647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:41495 every 8 connection(s)
I20260808 04:54:41.999898 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:54:42.003779 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 13437
I20260808 04:54:42.003947 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:54:42.004302 15481 external_mini_cluster.cc:1428] Setting key 18ab0d48752c21a305fb9bf5dbdc3d30
Aug 08 04:54:42 dist-test-slave-zpvx krb5kdc[13257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164881, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.65@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.66@KRBTEST.COM; defaulting to no policy
I20260808 04:54:42.014645 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.001716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:43123 (local address 127.15.30.126:35651)
0808 04:54:42.001853 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.001858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.003009 (+ 1151us) server_negotiation.cc:407] Connection header received
0808 04:54:42.003924 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.003927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.003991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.004086 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:42.005449 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.006132 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.007883 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.008305 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.008458 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.011040 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:42.011049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:42.011051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:42.011074 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:42.012615 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.013132 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.013135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.013137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.013216 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.013549 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.013554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.013556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.013748 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:42.013859 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.014345 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.014475 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:54:42.015424 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.001995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35651 (local address 127.15.30.65:43123)
0808 04:54:42.002857 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.002901 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.003711 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.004328 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.004340 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.004842 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:42.005302 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.005315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.006289 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.006295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.007746 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.007760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.008508 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.008511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.008693 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.009320 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:42.009336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:42.010856 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:42.012736 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:42.012743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:42.012756 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:42.013010 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:42.013310 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:42.013314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:42.013316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:42.013453 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:42.013859 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:42.013865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:42.014095 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.014295 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.014593 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":530,"threads_started":1}
I20260808 04:54:42.016690 13648 heartbeater.cc:347] Connected to a master server at 127.15.30.126:35651
I20260808 04:54:42.016956 13648 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.66@KRBTEST.COM" created.
I20260808 04:54:42.017510 13648 heartbeater.cc:511] Master 127.15.30.126:35651 requested a full tablet report, sending...
I20260808 04:54:42.019266 13334 ts_manager.cc:194] Registered new tserver with Master: d426339587eb42c9ab2ba530dea569d7 (127.15.30.65:41495)
I20260808 04:54:42.020716 13334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.65@KRBTEST.COM'} at 127.15.30.65:43123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
Entry for principal kudu/127.15.30.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab.
I20260808 04:54:42.033257 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35651
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:42.143934 13654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:42.144170 13654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:42.144212 13654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:42.148149 13654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:42.148207 13654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:42.148262 13654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:54:42.153354 13654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.66.keytab
--principal=kudu/127.15.30.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:42.154276 13654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:42.155210 13654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:42.164659 13666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:42.164673 13663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:42.164711 13662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:42.164908 13654 server_base.cc:1034] running on GCE node
I20260808 04:54:42.165311 13654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:42.165899 13654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:42.167050 13654 hybrid_clock.cc:648] HybridClock initialized: now 1786164882167033 us; error 26 us; skew 500 ppm
Aug 08 04:54:42 dist-test-slave-zpvx krb5kdc[13257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164882, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:42.169858 13654 init.cc:377] Logged in from keytab as kudu/127.15.30.66@KRBTEST.COM (short username kudu)
I20260808 04:54:42.170758 13654 webserver.cc:460] Webserver started at http://127.15.30.66:38777/ using document root <none> and password file <none>
I20260808 04:54:42.171310 13654 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:42.171402 13654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:42.171638 13654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:42.173722 13654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "ddad869e30fe4499aeab4bdd8940ae3e"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "dd67869904a16b838345b4c7f7ccdb7b"
server_key_iv: "f5ff5a4946e62683127b503e034917e4"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.174278 13654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "ddad869e30fe4499aeab4bdd8940ae3e"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "dd67869904a16b838345b4c7f7ccdb7b"
server_key_iv: "f5ff5a4946e62683127b503e034917e4"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.177820 13654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:54:42.180181 13674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:42.181409 13654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:42.181553 13654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "ddad869e30fe4499aeab4bdd8940ae3e"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "dd67869904a16b838345b4c7f7ccdb7b"
server_key_iv: "f5ff5a4946e62683127b503e034917e4"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.181665 13654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:42.195207 13654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:42.197960 13654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:42.198159 13654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:42.198720 13654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:42.199541 13654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:42.199615 13654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:42.199679 13654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:42.199724 13654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:42.208626 13654 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:39701
I20260808 04:54:42.208658 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:39701 every 8 connection(s)
I20260808 04:54:42.209597 13654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
Aug 08 04:54:42 dist-test-slave-zpvx krb5kdc[13257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164882, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.66@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:42.219249 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 13654
I20260808 04:54:42.219384 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:54:42.219676 15481 external_mini_cluster.cc:1428] Setting key f74dacb32e8b41a9a96f9eeddde6f151
I20260808 04:54:42.223407 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.211573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:51371 (local address 127.15.30.126:35651)
0808 04:54:42.211703 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.211706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.212549 (+ 843us) server_negotiation.cc:407] Connection header received
0808 04:54:42.213767 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.213770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.213818 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.213893 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:42.215088 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.215608 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.216741 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.216961 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.217042 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.219508 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:42.219511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:42.219513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:42.219538 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:42.221282 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.221772 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.221778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.221782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.221875 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.222198 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.222204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.222208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.222469 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:42.222575 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.223020 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.223162 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:54:42.224226 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.211926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35651 (local address 127.15.30.66:51371)
0808 04:54:42.212358 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.212451 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.213578 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.214052 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.214058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.214489 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:42.214961 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.214974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.215712 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.215714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.216638 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.216648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.217087 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.217089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.217219 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.217790 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:42.217805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:42.219302 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:42.221399 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:42.221403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:42.221414 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:42.221665 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:42.221971 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:42.221973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:42.221974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:42.222102 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:42.222571 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:42.222576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:42.222788 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.222941 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.223171 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260808 04:54:42.225407 13863 heartbeater.cc:347] Connected to a master server at 127.15.30.126:35651
I20260808 04:54:42.225648 13863 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:42.226115 13863 heartbeater.cc:511] Master 127.15.30.126:35651 requested a full tablet report, sending...
I20260808 04:54:42.227169 13334 ts_manager.cc:194] Registered new tserver with Master: ddad869e30fe4499aeab4bdd8940ae3e (127.15.30.66:39701)
I20260808 04:54:42.227738 13334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.66@KRBTEST.COM'} at 127.15.30.66:51371
WARNING: no policy specified for kudu/127.15.30.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
Entry for principal kudu/127.15.30.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab.
I20260808 04:54:42.249238 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35651
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:42.357052 13868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:42.357283 13868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:42.357323 13868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:42.361140 13868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:42.361182 13868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:42.361236 13868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:54:42.366317 13868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.67.keytab
--principal=kudu/127.15.30.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:42.367142 13868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:42.368103 13868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:42.377193 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:42.377250 13879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:42.377414 13868 server_base.cc:1034] running on GCE node
W20260808 04:54:42.377252 13880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:42.377983 13868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:42.378569 13868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:42.379755 13868 hybrid_clock.cc:648] HybridClock initialized: now 1786164882379723 us; error 52 us; skew 500 ppm
Aug 08 04:54:42 dist-test-slave-zpvx krb5kdc[13257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164882, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:42.382486 13868 init.cc:377] Logged in from keytab as kudu/127.15.30.67@KRBTEST.COM (short username kudu)
I20260808 04:54:42.383332 13868 webserver.cc:460] Webserver started at http://127.15.30.67:36033/ using document root <none> and password file <none>
I20260808 04:54:42.383876 13868 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:42.383968 13868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:42.384196 13868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:42.386216 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "85df196f5a7a4c4a9755537c90c78867"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "0c55a5e85e10ed496b98151a501db7c6"
server_key_iv: "2686d08c97dfc03aed2d833c3819dbf7"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.386775 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "85df196f5a7a4c4a9755537c90c78867"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "0c55a5e85e10ed496b98151a501db7c6"
server_key_iv: "2686d08c97dfc03aed2d833c3819dbf7"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.390306 13868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:42.392566 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:42.393709 13868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:42.393851 13868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "85df196f5a7a4c4a9755537c90c78867"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "0c55a5e85e10ed496b98151a501db7c6"
server_key_iv: "2686d08c97dfc03aed2d833c3819dbf7"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.393957 13868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:42.409359 13868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:42.412765 13868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:42.412961 13868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:42.413520 13868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:42.414307 13868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:42.414379 13868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:42.414451 13868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:42.414505 13868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:42.431510 13868 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:36813
I20260808 04:54:42.431555 14085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:36813 every 8 connection(s)
I20260808 04:54:42.432592 13868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:54:42.435432 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 13868
I20260808 04:54:42.435588 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:54:42.435925 15481 external_mini_cluster.cc:1428] Setting key 267f8fc2743ac76341b23f307a379dec
Aug 08 04:54:42 dist-test-slave-zpvx krb5kdc[13257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164882, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.67@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:54:42.450392 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.434535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:57455 (local address 127.15.30.126:35651)
0808 04:54:42.434687 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.434692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.435839 (+ 1147us) server_negotiation.cc:407] Connection header received
0808 04:54:42.437828 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.437831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.437884 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.437975 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:42.439772 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.440307 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.442420 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.442766 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.442881 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.446194 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:42.446199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:42.446202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:42.446234 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:42.447667 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.448285 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.448288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.448290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.448449 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.448926 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.448931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.448933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.449217 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:42.449318 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.450069 (+ 751us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.450216 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:54:42.451493 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.435003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35651 (local address 127.15.30.67:57455)
0808 04:54:42.435657 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.435720 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.437561 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.438337 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.438349 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.438913 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:42.439611 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.439634 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.440451 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.440455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.442251 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.442267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.443070 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.443074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.443223 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.443858 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:54:42.443884 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:42.446023 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:42.447813 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:42.447819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:42.447834 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:42.448174 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:42.448566 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:54:42.448572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:54:42.448574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:54:42.448794 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:54:42.449399 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:42.449408 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:42.449752 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.450010 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.450380 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":156,"threads_started":1}
I20260808 04:54:42.454339 14088 heartbeater.cc:347] Connected to a master server at 127.15.30.126:35651
I20260808 04:54:42.454566 14088 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:42.455075 14088 heartbeater.cc:511] Master 127.15.30.126:35651 requested a full tablet report, sending...
I20260808 04:54:42.456178 13334 ts_manager.cc:194] Registered new tserver with Master: 85df196f5a7a4c4a9755537c90c78867 (127.15.30.67:36813)
I20260808 04:54:42.456933 13334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.30.67@KRBTEST.COM'} at 127.15.30.67:57455
I20260808 04:54:42.462002 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:54:42.473663 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.464987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.15.30.126:35651)
0808 04:54:42.465101 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.465105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.465118 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:54:42.465175 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.465178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.465217 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.465320 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:42.466315 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.466866 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.468271 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.468563 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.468647 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.470429 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:42.470432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:42.470434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:42.470455 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:42.471883 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.472467 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.472470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.472471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.472544 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:42.472833 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:42.472835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:42.472836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:42.473032 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:42.473128 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.473439 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.473533 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 04:54:42.476630 13334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:54:42.479758 13334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:54:42.494477 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.488089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:41495 (local address 127.0.0.1:42282)
0808 04:54:42.488701 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.488748 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.488873 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.490357 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.490361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.490380 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:42.490479 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.490486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.491892 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.491896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.492869 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.492880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.493867 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.493870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.494058 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.494084 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.494184 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.494330 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":51,"thread_start_us":82,"threads_started":1}
I20260808 04:54:42.494481 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.488211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.15.30.65:41495)
0808 04:54:42.489890 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.489905 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.489927 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:54:42.490004 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.490008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.490176 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.490629 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:42.490787 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.491767 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.493003 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.493711 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.494025 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.494175 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.494229 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.494335 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559,"thread_start_us":106,"threads_started":1}
I20260808 04:54:42.494530 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.488965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.15.30.67:36813)
0808 04:54:42.489673 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.489681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.489700 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:54:42.489761 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.489764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.489949 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.490099 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:42.490482 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.491217 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.492951 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.493669 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.493789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.493929 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.494308 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.494395 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":159,"threads_started":1}
I20260808 04:54:42.495007 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.488579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:36813 (local address 127.0.0.1:50530)
0808 04:54:42.489543 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.489555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.489637 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.490276 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.490278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.490294 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:42.490357 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.490361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.491353 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.491358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.492817 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.492828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.493819 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.493824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.494005 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.494080 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.494196 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.494869 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":648,"thread_start_us":90,"threads_started":1}
I20260808 04:54:42.496161 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.488187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:39701 (local address 127.0.0.1:47744)
0808 04:54:42.489295 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.489311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.489430 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.489941 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.489947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.489971 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:42.490122 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.490129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.492060 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.492063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.493265 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.493276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.494456 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.494459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.495840 (+ 1381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.495868 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.495970 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.496040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":44,"thread_start_us":112,"threads_started":1}
I20260808 04:54:42.496244 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.488882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.15.30.66:39701)
0808 04:54:42.489250 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.489266 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.489419 (+ 153us) server_negotiation.cc:407] Connection header received
0808 04:54:42.489545 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.489554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.489770 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.489934 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:42.490259 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.491427 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.493392 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.494164 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.494274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.494394 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.496005 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.496095 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260808 04:54:42.498015 13977 tablet_service.cc:1467] Processing CreateTablet for tablet 2511af774c644ac2b0fc209ccb4ee407 (DEFAULT_TABLE table=test-table [id=c950e11f846642fe8899f76743eba975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:42.498015 13540 tablet_service.cc:1467] Processing CreateTablet for tablet 2511af774c644ac2b0fc209ccb4ee407 (DEFAULT_TABLE table=test-table [id=c950e11f846642fe8899f76743eba975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:42.499187 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2511af774c644ac2b0fc209ccb4ee407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:42.499191 13977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2511af774c644ac2b0fc209ccb4ee407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:42.502928 13752 tablet_service.cc:1467] Processing CreateTablet for tablet 2511af774c644ac2b0fc209ccb4ee407 (DEFAULT_TABLE table=test-table [id=c950e11f846642fe8899f76743eba975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:42.503942 13752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2511af774c644ac2b0fc209ccb4ee407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:42.506817 14117 tablet_bootstrap.cc:492] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Bootstrap starting.
I20260808 04:54:42.507241 14119 tablet_bootstrap.cc:492] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7: Bootstrap starting.
I20260808 04:54:42.509639 14119 tablet_bootstrap.cc:654] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:42.510388 14117 tablet_bootstrap.cc:654] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:42.511015 14119 log.cc:824] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:42.511461 14120 tablet_bootstrap.cc:492] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Bootstrap starting.
I20260808 04:54:42.511595 14117 log.cc:824] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:42.513087 14119 tablet_bootstrap.cc:492] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7: No bootstrap required, opened a new log
I20260808 04:54:42.513283 14119 ts_tablet_manager.cc:1397] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:54:42.513607 14117 tablet_bootstrap.cc:492] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: No bootstrap required, opened a new log
I20260808 04:54:42.513797 14117 ts_tablet_manager.cc:1397] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:54:42.514387 14120 tablet_bootstrap.cc:654] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:42.515053 14120 log.cc:824] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:42.516819 14120 tablet_bootstrap.cc:492] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: No bootstrap required, opened a new log
I20260808 04:54:42.517024 14120 ts_tablet_manager.cc:1397] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:54:42.517241 14119 raft_consensus.cc:348] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.517508 14119 raft_consensus.cc:374] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:42.517573 14119 raft_consensus.cc:729] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d426339587eb42c9ab2ba530dea569d7, State: Initialized, Role: FOLLOWER
I20260808 04:54:42.517467 14117 raft_consensus.cc:348] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.517731 14117 raft_consensus.cc:374] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:42.517789 14117 raft_consensus.cc:729] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85df196f5a7a4c4a9755537c90c78867, State: Initialized, Role: FOLLOWER
I20260808 04:54:42.518054 14119 consensus_queue.cc:260] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.518148 14117 consensus_queue.cc:260] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.518785 14088 heartbeater.cc:503] Master 127.15.30.126:35651 was elected leader, sending a full tablet report...
I20260808 04:54:42.519012 14117 ts_tablet_manager.cc:1428] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:54:42.519392 13648 heartbeater.cc:503] Master 127.15.30.126:35651 was elected leader, sending a full tablet report...
I20260808 04:54:42.519672 14119 ts_tablet_manager.cc:1428] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:54:42.520859 14120 raft_consensus.cc:348] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.521128 14120 raft_consensus.cc:374] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:42.521207 14120 raft_consensus.cc:729] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddad869e30fe4499aeab4bdd8940ae3e, State: Initialized, Role: FOLLOWER
I20260808 04:54:42.521672 14120 consensus_queue.cc:260] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.522449 13863 heartbeater.cc:503] Master 127.15.30.126:35651 was elected leader, sending a full tablet report...
I20260808 04:54:42.523552 14120 ts_tablet_manager.cc:1428] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 04:54:42.683571 14089 tablet.cc:2369] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:42.689373 14129 raft_consensus.cc:482] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:42.689589 14129 raft_consensus.cc:504] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.691434 14129 leader_election.cc:288] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d426339587eb42c9ab2ba530dea569d7 (127.15.30.65:41495), 85df196f5a7a4c4a9755537c90c78867 (127.15.30.67:36813)
I20260808 04:54:42.698143 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.692811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:41495 (local address 127.15.30.66:43753)
0808 04:54:42.693408 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.693424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.693604 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.693902 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.693905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.693922 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:42.694052 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.694057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.694776 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.694782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.696408 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.696420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.697630 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.697635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.697760 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.697783 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.697912 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.697986 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":104,"threads_started":1}
I20260808 04:54:42.698139 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.691751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:43753 (local address 127.15.30.65:41495)
0808 04:54:42.691870 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.691874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.693519 (+ 1645us) server_negotiation.cc:407] Connection header received
0808 04:54:42.693730 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.693733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.693796 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.693882 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:42.694167 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.694659 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.696574 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.697317 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.697433 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.697511 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.697932 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.698017 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:54:42.699136 13571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511af774c644ac2b0fc209ccb4ee407" candidate_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d426339587eb42c9ab2ba530dea569d7" is_pre_election: true
I20260808 04:54:42.699342 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.691297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:36813 (local address 127.15.30.66:45573)
0808 04:54:42.691569 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.691585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:42.692151 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:42.692535 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:42.692538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:42.692555 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:42.692664 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.692670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.693998 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.694001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.695243 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:42.695254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.698947 (+ 3693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.698950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:42.699052 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.699076 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:54:42.699158 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:54:42.699226 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 04:54:42.699474 13571 raft_consensus.cc:2454] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddad869e30fe4499aeab4bdd8940ae3e in term 0.
I20260808 04:54:42.700646 13679 leader_election.cc:302] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d426339587eb42c9ab2ba530dea569d7, ddad869e30fe4499aeab4bdd8940ae3e; no voters:
I20260808 04:54:42.700875 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.691930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:45573 (local address 127.15.30.67:36813)
0808 04:54:42.692036 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.692040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.692231 (+ 191us) server_negotiation.cc:407] Connection header received
0808 04:54:42.692285 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.692288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.692332 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.692423 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:42.693076 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.693862 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.695426 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.696067 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.698943 (+ 2876us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.698992 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.700590 (+ 1598us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.700704 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:54:42.700959 14129 raft_consensus.cc:2781] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:54:42.701036 14129 raft_consensus.cc:482] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:54:42.701082 14129 raft_consensus.cc:3037] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:42.702636 14129 raft_consensus.cc:504] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.702831 14001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511af774c644ac2b0fc209ccb4ee407" candidate_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85df196f5a7a4c4a9755537c90c78867" is_pre_election: true
I20260808 04:54:42.703024 14129 leader_election.cc:288] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [CANDIDATE]: Term 1 election: Requested vote from peers d426339587eb42c9ab2ba530dea569d7 (127.15.30.65:41495), 85df196f5a7a4c4a9755537c90c78867 (127.15.30.67:36813)
I20260808 04:54:42.703194 14001 raft_consensus.cc:2454] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddad869e30fe4499aeab4bdd8940ae3e in term 0.
I20260808 04:54:42.703720 14005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511af774c644ac2b0fc209ccb4ee407" candidate_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85df196f5a7a4c4a9755537c90c78867"
I20260808 04:54:42.703847 14005 raft_consensus.cc:3037] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:42.703871 13571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2511af774c644ac2b0fc209ccb4ee407" candidate_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d426339587eb42c9ab2ba530dea569d7"
I20260808 04:54:42.704002 13571 raft_consensus.cc:3037] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:42.705235 14005 raft_consensus.cc:2454] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddad869e30fe4499aeab4bdd8940ae3e in term 1.
I20260808 04:54:42.705463 13571 raft_consensus.cc:2454] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddad869e30fe4499aeab4bdd8940ae3e in term 1.
I20260808 04:54:42.705540 13676 leader_election.cc:302] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85df196f5a7a4c4a9755537c90c78867, ddad869e30fe4499aeab4bdd8940ae3e; no voters:
I20260808 04:54:42.705766 14129 raft_consensus.cc:2781] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:42.706001 14129 raft_consensus.cc:686] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 1 LEADER]: Becoming Leader. State: Replica: ddad869e30fe4499aeab4bdd8940ae3e, State: Running, Role: LEADER
I20260808 04:54:42.706367 14129 consensus_queue.cc:237] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } }
I20260808 04:54:42.709772 13330 catalog_manager.cc:5495] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e reported cstate change: term changed from 0 to 1, leader changed from <none> to ddad869e30fe4499aeab4bdd8940ae3e (127.15.30.66). New cstate: current_term: 1 leader_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddad869e30fe4499aeab4bdd8940ae3e" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 39701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 } health_report { overall_health: UNKNOWN } } }
W20260808 04:54:42.710776 13864 tablet.cc:2369] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:42.730525 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.726939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.15.30.66:39701)
0808 04:54:42.727061 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:42.727067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:42.727119 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:54:42.727300 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:42.727303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:42.727357 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:42.727467 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:54:42.727712 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.728214 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.729110 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:42.729330 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:42.729443 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:42.729751 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:54:42.730118 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:54:42.730240 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:42.730321 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:54:42.730375 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:54:42.736029 13571 raft_consensus.cc:1261] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 1 FOLLOWER]: Refusing update from remote peer ddad869e30fe4499aeab4bdd8940ae3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:42.736042 14005 raft_consensus.cc:1261] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 1 FOLLOWER]: Refusing update from remote peer ddad869e30fe4499aeab4bdd8940ae3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:42.736771 14131 consensus_queue.cc:1035] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "85df196f5a7a4c4a9755537c90c78867" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:42.736929 14129 consensus_queue.cc:1035] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d426339587eb42c9ab2ba530dea569d7" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:42.747298 14135 mvcc.cc:205] Tried to move back new op lower bound from 7316131359679672320 to 7316131359568216064. Current Snapshot: MvccSnapshot[applied={T|T < 7316131359679672320 or (T in {7316131359679672320})}]
I20260808 04:54:42.751609 13334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:42.751777 13334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:42.754125 13334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 002c34fab3bb44cb8d98eeed3396163b: Sending DeleteTablet for 3 replicas of tablet 2511af774c644ac2b0fc209ccb4ee407
I20260808 04:54:42.754827 13540 tablet_service.cc:1514] Processing DeleteTablet for tablet 2511af774c644ac2b0fc209ccb4ee407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:42 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:42282
I20260808 04:54:42.754974 13977 tablet_service.cc:1514] Processing DeleteTablet for tablet 2511af774c644ac2b0fc209ccb4ee407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:42 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:50530
I20260808 04:54:42.755194 13752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2511af774c644ac2b0fc209ccb4ee407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:42 UTC) from {username='kudu', principal='kudu/127.15.30.126@KRBTEST.COM'} at 127.0.0.1:47744
I20260808 04:54:42.755307 14148 tablet_replica.cc:330] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7: stopping tablet replica
I20260808 04:54:42.755373 14149 tablet_replica.cc:330] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: stopping tablet replica
I20260808 04:54:42.755579 14149 raft_consensus.cc:2229] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:42.755579 14148 raft_consensus.cc:2229] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:42.755851 14148 raft_consensus.cc:2258] T 2511af774c644ac2b0fc209ccb4ee407 P d426339587eb42c9ab2ba530dea569d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:42.755851 14149 raft_consensus.cc:2258] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:42.756247 14152 tablet_replica.cc:330] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: stopping tablet replica
I20260808 04:54:42.756353 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 13437
I20260808 04:54:42.765275 14149 ts_tablet_manager.cc:1905] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:42.768476 14149 ts_tablet_manager.cc:1918] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:42.768572 14149 log.cc:1196] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/2511af774c644ac2b0fc209ccb4ee407
I20260808 04:54:42.768869 14149 ts_tablet_manager.cc:1939] T 2511af774c644ac2b0fc209ccb4ee407 P 85df196f5a7a4c4a9755537c90c78867: Deleting consensus metadata
I20260808 04:54:42.770061 13305 catalog_manager.cc:4841] TS 85df196f5a7a4c4a9755537c90c78867 (127.15.30.67:36813): tablet 2511af774c644ac2b0fc209ccb4ee407 (table test-table [id=c950e11f846642fe8899f76743eba975]) successfully deleted
I20260808 04:54:42.775923 14152 raft_consensus.cc:2229] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:54:42.776283 14152 raft_consensus.cc:2258] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:42.777848 14152 ts_tablet_manager.cc:1905] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:42.779949 14152 ts_tablet_manager.cc:1918] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:42.780071 14152 log.cc:1196] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/2511af774c644ac2b0fc209ccb4ee407
I20260808 04:54:42.780453 14152 ts_tablet_manager.cc:1939] T 2511af774c644ac2b0fc209ccb4ee407 P ddad869e30fe4499aeab4bdd8940ae3e: Deleting consensus metadata
I20260808 04:54:42.781514 13311 catalog_manager.cc:4841] TS ddad869e30fe4499aeab4bdd8940ae3e (127.15.30.66:39701): tablet 2511af774c644ac2b0fc209ccb4ee407 (table test-table [id=c950e11f846642fe8899f76743eba975]) successfully deleted
W20260808 04:54:42.784560 13306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:41495 (error 108)
I20260808 04:54:42.784718 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 13654
I20260808 04:54:42.790685 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:42.790398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:41495 (local address 127.0.0.1:42286)
0808 04:54:42.790537 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:42.790608 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:41495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 04:54:42.790952 13306 catalog_manager.cc:4584] TS d426339587eb42c9ab2ba530dea569d7 (127.15.30.65:41495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2511af774c644ac2b0fc209ccb4ee407: Network error: Client connection negotiation failed: client connection to 127.15.30.65:41495: connect: Connection refused (error 111)
I20260808 04:54:42.795459 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 13868
I20260808 04:54:42.808276 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 13273
2026-08-08T04:54:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2670 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T04:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:42Z Disabled control of system clock
I20260808 04:54:42.844974 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35217
--webserver_interface=127.15.30.126
--webserver_port=0
--builtin_ntp_servers=127.15.30.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:54:42.955756 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:42.956019 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:42.956053 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:42.959971 14159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:42.960023 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:42.960039 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:42.960055 14159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:42.960070 14159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:42.964650 14159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:35217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:42.965626 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:42.967031 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:42.975694 14171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:42.975710 14168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:42.975646 14176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:42.975874 14159 server_base.cc:1034] running on GCE node
I20260808 04:54:42.976488 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:42.977517 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:42.978696 14159 hybrid_clock.cc:648] HybridClock initialized: now 1786164882978670 us; error 37 us; skew 500 ppm
I20260808 04:54:42.986447 14159 webserver.cc:460] Webserver started at http://127.15.30.126:39931/ using document root <none> and password file <none>
I20260808 04:54:42.987071 14159 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:42.987134 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:42.987376 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:42.989540 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "8e796827141f4fe2aede99006f3866f0"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "93038f8d06351943a1dbf0a5b7b57343"
server_key_iv: "083d711d87a650bacfeb2c0b9b72f388"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.990135 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "8e796827141f4fe2aede99006f3866f0"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "93038f8d06351943a1dbf0a5b7b57343"
server_key_iv: "083d711d87a650bacfeb2c0b9b72f388"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.993783 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:54:42.996194 14186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:42.997584 14159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:42.997735 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "8e796827141f4fe2aede99006f3866f0"
format_stamp: "Formatted at 2026-08-08 04:54:42 on dist-test-slave-zpvx"
server_key: "93038f8d06351943a1dbf0a5b7b57343"
server_key_iv: "083d711d87a650bacfeb2c0b9b72f388"
server_key_version: "encryptionkey@0"
I20260808 04:54:42.997857 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:43.009459 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:43.010097 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:43.010290 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:43.017609 14159 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:35217
I20260808 04:54:43.017637 14279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:35217 every 8 connection(s)
I20260808 04:54:43.018509 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:43.020648 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 14159
I20260808 04:54:43.020790 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:43.021121 15481 external_mini_cluster.cc:1428] Setting key b929a5a72c1f33698bf1da8f9d9f5969
I20260808 04:54:43.021895 14281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:43.028080 14281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0: Bootstrap starting.
I20260808 04:54:43.029982 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.022394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.15.30.126:35217)
0808 04:54:43.023024 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.023040 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.023076 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:54:43.023714 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.023729 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.024107 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.024476 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:43.024897 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.025857 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.026762 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.027216 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.027397 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.028259 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:43.028263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:43.028278 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:43.028703 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:43.028757 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:43.028786 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:43.028896 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.029070 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.029312 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":125,"threads_started":1}
I20260808 04:54:43.030416 14281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:43.031257 14281 log.cc:824] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:43.033161 14281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0: No bootstrap required, opened a new log
I20260808 04:54:43.036741 14281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e796827141f4fe2aede99006f3866f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35217 } }
I20260808 04:54:43.036935 14281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:43.036968 14281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e796827141f4fe2aede99006f3866f0, State: Initialized, Role: FOLLOWER
I20260808 04:54:43.037344 14281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e796827141f4fe2aede99006f3866f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35217 } }
I20260808 04:54:43.037461 14281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:43.037506 14281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:43.037576 14281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:43.038324 14281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e796827141f4fe2aede99006f3866f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35217 } }
I20260808 04:54:43.038710 14281 leader_election.cc:302] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e796827141f4fe2aede99006f3866f0; no voters:
I20260808 04:54:43.039062 14281 leader_election.cc:288] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:43.039134 14291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:43.039378 14291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8e796827141f4fe2aede99006f3866f0, State: Running, Role: LEADER
I20260808 04:54:43.039693 14291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e796827141f4fe2aede99006f3866f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35217 } }
I20260808 04:54:43.040202 14281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:43.041390 14296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e796827141f4fe2aede99006f3866f0. Latest consensus state: current_term: 1 leader_uuid: "8e796827141f4fe2aede99006f3866f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e796827141f4fe2aede99006f3866f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35217 } } }
I20260808 04:54:43.041390 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e796827141f4fe2aede99006f3866f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e796827141f4fe2aede99006f3866f0" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35217 } } }
I20260808 04:54:43.041548 14296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:43.041548 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:43.041980 14308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:43.044642 14308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:43.050240 14308 catalog_manager.cc:1347] Generated new cluster ID: 43a8abfde3c3467b8a5c99d063efff6e
I20260808 04:54:43.050318 14308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:43.066591 14308 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:43.067440 14308 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:43.075156 14308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0: Generated new TSK 0
I20260808 04:54:43.075744 14308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:43.082408 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.65:0
--local_ip_for_outbound_sockets=127.15.30.65
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35217
--builtin_ntp_servers=127.15.30.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:54:43.194749 14328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:43.194993 14328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:43.195032 14328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:43.199090 14328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:43.199149 14328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:43.199244 14328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.65
I20260808 04:54:43.204155 14328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.30.65
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.65
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:43.205214 14328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:43.206593 14328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:43.216025 14339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:43.216035 14340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:43.216035 14342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:43.216354 14328 server_base.cc:1034] running on GCE node
I20260808 04:54:43.216738 14328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:43.217291 14328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:43.218478 14328 hybrid_clock.cc:648] HybridClock initialized: now 1786164883218461 us; error 29 us; skew 500 ppm
I20260808 04:54:43.220266 14328 webserver.cc:460] Webserver started at http://127.15.30.65:44487/ using document root <none> and password file <none>
I20260808 04:54:43.220860 14328 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:43.220954 14328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:43.221189 14328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:43.223165 14328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data/instance:
uuid: "51604b193f214bf68dc8e14a473e65a9"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "91b20209ac78f48b9bba5f7f7738f3af"
server_key_iv: "2cae5d5b921c5a6edb753f1994e5dca0"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.223743 14328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance:
uuid: "51604b193f214bf68dc8e14a473e65a9"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "91b20209ac78f48b9bba5f7f7738f3af"
server_key_iv: "2cae5d5b921c5a6edb753f1994e5dca0"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.227280 14328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260808 04:54:43.229646 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.230758 14328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:54:43.230891 14328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/wal
uuid: "51604b193f214bf68dc8e14a473e65a9"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "91b20209ac78f48b9bba5f7f7738f3af"
server_key_iv: "2cae5d5b921c5a6edb753f1994e5dca0"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.230998 14328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:43.244127 14328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:43.244799 14328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:43.244989 14328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:43.245543 14328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:43.246450 14328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:43.246522 14328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.246594 14328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:43.246646 14328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.255262 14328 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.65:39803
I20260808 04:54:43.255286 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.65:39803 every 8 connection(s)
I20260808 04:54:43.256099 14328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/data/info.pb
I20260808 04:54:43.259590 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 14328
I20260808 04:54:43.259737 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-0/wal/instance
I20260808 04:54:43.260107 15481 external_mini_cluster.cc:1428] Setting key bb9828238652dea1b19075555d12d985
I20260808 04:54:43.262197 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.66:0
--local_ip_for_outbound_sockets=127.15.30.66
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35217
--builtin_ntp_servers=127.15.30.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:54:43.265875 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.257898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.65:42921 (local address 127.15.30.126:35217)
0808 04:54:43.258031 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.258039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.258994 (+ 955us) server_negotiation.cc:407] Connection header received
0808 04:54:43.260847 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.260851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.260915 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.260998 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:43.262044 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.262703 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.263852 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.264101 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.264212 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.265357 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:43.265361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:43.265363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:43.265376 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:43.265397 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:43.265413 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:43.265491 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.265651 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.265723 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:54:43.266651 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.258229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35217 (local address 127.15.30.65:42921)
0808 04:54:43.258849 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.258886 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.260633 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.261180 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.261186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.261277 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:43.261875 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.261895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.262843 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.262845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.263715 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.263724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.264263 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.264268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.264487 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.265097 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:43.265099 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:43.265137 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:43.265198 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:43.265504 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:43.265518 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:43.265527 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.265650 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.265912 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260808 04:54:43.267841 14558 heartbeater.cc:347] Connected to a master server at 127.15.30.126:35217
I20260808 04:54:43.268126 14558 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:43.268812 14558 heartbeater.cc:511] Master 127.15.30.126:35217 requested a full tablet report, sending...
I20260808 04:54:43.270632 14213 ts_manager.cc:194] Registered new tserver with Master: 51604b193f214bf68dc8e14a473e65a9 (127.15.30.65:39803)
I20260808 04:54:43.272440 14213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.65:42921
W20260808 04:54:43.375628 14562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:43.375852 14562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:43.375887 14562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:43.379809 14562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:43.379859 14562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:43.379910 14562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.66
I20260808 04:54:43.384479 14562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.30.66
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.66
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:43.385373 14562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:43.386621 14562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:43.395854 14570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:43.395915 14562 server_base.cc:1034] running on GCE node
W20260808 04:54:43.395862 14574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:43.395869 14569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:43.396536 14562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:43.397188 14562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:43.398375 14562 hybrid_clock.cc:648] HybridClock initialized: now 1786164883398347 us; error 41 us; skew 500 ppm
I20260808 04:54:43.400133 14562 webserver.cc:460] Webserver started at http://127.15.30.66:42883/ using document root <none> and password file <none>
I20260808 04:54:43.400763 14562 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:43.400866 14562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:43.401176 14562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:43.403314 14562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data/instance:
uuid: "a45620bd03524564820f94f45f18b49f"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "263594372dc344515aeafdc205682bd0"
server_key_iv: "6f85c23fd277b88ca388f170df0a100f"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.403888 14562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance:
uuid: "a45620bd03524564820f94f45f18b49f"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "263594372dc344515aeafdc205682bd0"
server_key_iv: "6f85c23fd277b88ca388f170df0a100f"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.407440 14562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 04:54:43.409863 14585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.410953 14562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 04:54:43.411088 14562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal
uuid: "a45620bd03524564820f94f45f18b49f"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "263594372dc344515aeafdc205682bd0"
server_key_iv: "6f85c23fd277b88ca388f170df0a100f"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.411198 14562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:43.428319 14562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:43.428994 14562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:43.429201 14562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:43.429757 14562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:43.430666 14562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:43.430742 14562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.430824 14562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:43.430862 14562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.439708 14562 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.66:37389
I20260808 04:54:43.439754 14777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.66:37389 every 8 connection(s)
I20260808 04:54:43.440595 14562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/data/info.pb
I20260808 04:54:43.449236 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.442457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.66:34035 (local address 127.15.30.126:35217)
0808 04:54:43.442601 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.442605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.443571 (+ 966us) server_negotiation.cc:407] Connection header received
0808 04:54:43.444992 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.444994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.445044 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.445117 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:43.445956 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.446455 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.447520 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.447722 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.447802 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.448811 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:43.448814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:43.448817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:43.448828 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:43.448844 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:43.448852 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:43.448909 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.449056 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.449124 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:54:43.449352 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 14562
I20260808 04:54:43.449493 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal/instance
I20260808 04:54:43.449820 15481 external_mini_cluster.cc:1428] Setting key 0c1fbe1d07e96e7b70c0d7e82f4201fa
I20260808 04:54:43.450302 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.442788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35217 (local address 127.15.30.66:34035)
0808 04:54:43.443412 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.443451 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.444792 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.445294 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.445300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.445382 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:43.445831 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.445843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.446562 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.446564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.447411 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.447421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.447858 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.447861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.448002 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.448561 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:43.448562 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:43.448600 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:43.448660 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:43.448926 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:43.448932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:43.448937 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.449040 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.449342 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":119,"threads_started":1}
I20260808 04:54:43.451511 14778 heartbeater.cc:347] Connected to a master server at 127.15.30.126:35217
I20260808 04:54:43.451735 14778 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:43.452001 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.30.67:0
--local_ip_for_outbound_sockets=127.15.30.67
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35217
--builtin_ntp_servers=127.15.30.84:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:54:43.452284 14778 heartbeater.cc:511] Master 127.15.30.126:35217 requested a full tablet report, sending...
I20260808 04:54:43.453322 14213 ts_manager.cc:194] Registered new tserver with Master: a45620bd03524564820f94f45f18b49f (127.15.30.66:37389)
I20260808 04:54:43.454007 14213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.66:34035
W20260808 04:54:43.565655 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:43.565877 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:43.565912 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:43.569748 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:43.569790 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:43.569840 14785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.30.67
I20260808 04:54:43.574484 14785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.30.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.30.67
--webserver_port=0
--tserver_master_addrs=127.15.30.126:35217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.30.67
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:43.575589 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:43.577574 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:43.587111 14796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:43.587201 14785 server_base.cc:1034] running on GCE node
W20260808 04:54:43.587121 14794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:43.587075 14800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:43.587702 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:43.588330 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:43.589540 14785 hybrid_clock.cc:648] HybridClock initialized: now 1786164883589510 us; error 42 us; skew 500 ppm
I20260808 04:54:43.591251 14785 webserver.cc:460] Webserver started at http://127.15.30.67:36901/ using document root <none> and password file <none>
I20260808 04:54:43.591818 14785 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:43.591912 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:43.592141 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:43.594157 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data/instance:
uuid: "deec178cab0941a8bce243866a4cd77d"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "cfb6181401c0a7c328429f421bde7559"
server_key_iv: "2cc86fefc4f44117717e911f20f76af9"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.594725 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance:
uuid: "deec178cab0941a8bce243866a4cd77d"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "cfb6181401c0a7c328429f421bde7559"
server_key_iv: "2cc86fefc4f44117717e911f20f76af9"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.598325 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:43.600839 14808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.602126 14785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:43.602276 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal
uuid: "deec178cab0941a8bce243866a4cd77d"
format_stamp: "Formatted at 2026-08-08 04:54:43 on dist-test-slave-zpvx"
server_key: "cfb6181401c0a7c328429f421bde7559"
server_key_iv: "2cc86fefc4f44117717e911f20f76af9"
server_key_version: "encryptionkey@0"
I20260808 04:54:43.602387 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:43.619575 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:43.620221 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:43.620476 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:43.621042 14785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:54:43.621953 14785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:54:43.622027 14785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.622100 14785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:54:43.622144 14785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:43.630612 14785 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.67:36061
I20260808 04:54:43.630640 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.67:36061 every 8 connection(s)
I20260808 04:54:43.631491 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/data/info.pb
I20260808 04:54:43.640268 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 14785
I20260808 04:54:43.640426 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal/instance
I20260808 04:54:43.640792 15481 external_mini_cluster.cc:1428] Setting key e59c323e2bea8de90268b56831f45f73
I20260808 04:54:43.640879 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.633347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:57573 (local address 127.15.30.126:35217)
0808 04:54:43.633488 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.633492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.634531 (+ 1039us) server_negotiation.cc:407] Connection header received
0808 04:54:43.635979 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.635982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.636039 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.636150 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:43.637057 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.637593 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.638728 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.638953 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.639066 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.640221 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:43.640225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:43.640227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:43.640239 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:43.640258 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:43.640267 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:43.640373 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.640598 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.640700 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:54:43.641741 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.633729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35217 (local address 127.15.30.67:57573)
0808 04:54:43.634390 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.634429 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.635772 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.636311 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.636317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.636456 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:54:43.636916 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.636928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.637732 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.637736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.638593 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.638602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.639106 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.639110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.639270 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.639869 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:54:43.639870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:54:43.639909 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:54:43.639968 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:54:43.640388 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:54:43.640400 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:54:43.640408 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.640586 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.640882 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260808 04:54:43.642755 14999 heartbeater.cc:347] Connected to a master server at 127.15.30.126:35217
I20260808 04:54:43.643004 14999 heartbeater.cc:464] Registering TS with master...
I20260808 04:54:43.643507 14999 heartbeater.cc:511] Master 127.15.30.126:35217 requested a full tablet report, sending...
I20260808 04:54:43.644498 14213 ts_manager.cc:194] Registered new tserver with Master: deec178cab0941a8bce243866a4cd77d (127.15.30.67:36061)
I20260808 04:54:43.645054 14213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.30.67:57573
I20260808 04:54:43.654532 15481 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:54:43.659931 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.656412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.15.30.126:35217)
0808 04:54:43.656533 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.656537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.656715 (+ 178us) server_negotiation.cc:407] Connection header received
0808 04:54:43.656795 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.656797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.656834 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.656898 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:43.657144 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.657613 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.658416 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.658614 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.658689 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.659459 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:43.659461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:43.659463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:43.659473 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:43.659487 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:43.659493 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:43.659546 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.659729 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.659771 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:54:43.662760 14213 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:54:43.665827 14213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:54:43.679762 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.674207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.67:36061 (local address 127.0.0.1:58420)
0808 04:54:43.674597 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.674649 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.674799 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.675726 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.675734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.675765 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:43.675916 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.675925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.677203 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.677207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.678140 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.678152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.679213 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.679217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.679339 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.679383 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.679501 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.679612 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260808 04:54:43.679829 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.674374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.15.30.67:36061)
0808 04:54:43.674991 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.675024 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.675053 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:54:43.675135 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.675140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.675367 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.675533 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:43.676040 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.677091 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.678290 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.679069 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.679191 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.679384 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.679555 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.679661 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":167,"threads_started":1}
I20260808 04:54:43.680862 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.675279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:39803 (local address 127.0.0.1:38548)
0808 04:54:43.675488 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.675500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.675580 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.676598 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.676601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.676618 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:43.676697 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.676703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.678073 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.678076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.679101 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.679111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.680429 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.680434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.680528 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.680552 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.680612 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.680683 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260808 04:54:43.680893 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.675632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.15.30.65:39803)
0808 04:54:43.676117 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.676126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.676144 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:54:43.676206 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.676210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.676391 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.676578 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:43.676835 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.677937 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.679269 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.680271 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.680416 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.680592 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.680639 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.680719 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":140,"threads_started":1}
I20260808 04:54:43.681756 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.674954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:37389 (local address 127.0.0.1:37128)
0808 04:54:43.675179 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.675196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.675288 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.676424 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.676427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.676448 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:43.676570 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.676577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.678165 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.678168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.679419 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.679430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.680491 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.680495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.680564 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.680595 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.680640 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.680683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260808 04:54:43.682586 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.675130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.15.30.66:37389)
0808 04:54:43.675942 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.675955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.675980 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:54:43.676059 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.676064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.676252 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.676431 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:43.676705 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.677902 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.679586 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.680322 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.682106 (+ 1784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.682288 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.682340 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.682430 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":98,"threads_started":1}
I20260808 04:54:43.683141 14884 tablet_service.cc:1467] Processing CreateTablet for tablet a980119d8f7e4ec3ba7df0114ce174b0 (DEFAULT_TABLE table=test-table [id=812fff17c9a74369913ac0649452ff16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:43.684187 14884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a980119d8f7e4ec3ba7df0114ce174b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:43.683781 14432 tablet_service.cc:1467] Processing CreateTablet for tablet a980119d8f7e4ec3ba7df0114ce174b0 (DEFAULT_TABLE table=test-table [id=812fff17c9a74369913ac0649452ff16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:43.684854 14432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a980119d8f7e4ec3ba7df0114ce174b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:43.685372 14665 tablet_service.cc:1467] Processing CreateTablet for tablet a980119d8f7e4ec3ba7df0114ce174b0 (DEFAULT_TABLE table=test-table [id=812fff17c9a74369913ac0649452ff16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:54:43.686322 14665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a980119d8f7e4ec3ba7df0114ce174b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:43.690860 15027 tablet_bootstrap.cc:492] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Bootstrap starting.
I20260808 04:54:43.692206 15029 tablet_bootstrap.cc:492] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9: Bootstrap starting.
I20260808 04:54:43.693257 15027 tablet_bootstrap.cc:654] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:43.693523 15030 tablet_bootstrap.cc:492] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Bootstrap starting.
I20260808 04:54:43.694279 15027 log.cc:824] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:43.694365 15029 tablet_bootstrap.cc:654] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:43.695070 15029 log.cc:824] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:43.695658 15030 tablet_bootstrap.cc:654] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:43.696475 15027 tablet_bootstrap.cc:492] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: No bootstrap required, opened a new log
I20260808 04:54:43.696508 15030 log.cc:824] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:43.696667 15027 ts_tablet_manager.cc:1397] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:54:43.697571 15029 tablet_bootstrap.cc:492] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9: No bootstrap required, opened a new log
I20260808 04:54:43.697769 15029 ts_tablet_manager.cc:1397] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:54:43.698215 15030 tablet_bootstrap.cc:492] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: No bootstrap required, opened a new log
I20260808 04:54:43.698415 15030 ts_tablet_manager.cc:1397] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 04:54:43.701056 15027 raft_consensus.cc:348] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.701336 15027 raft_consensus.cc:374] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:43.701406 15027 raft_consensus.cc:729] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deec178cab0941a8bce243866a4cd77d, State: Initialized, Role: FOLLOWER
I20260808 04:54:43.701516 15029 raft_consensus.cc:348] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.701756 15029 raft_consensus.cc:374] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:43.701809 15029 raft_consensus.cc:729] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51604b193f214bf68dc8e14a473e65a9, State: Initialized, Role: FOLLOWER
I20260808 04:54:43.701738 15030 raft_consensus.cc:348] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.701853 15027 consensus_queue.cc:260] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.701995 15030 raft_consensus.cc:374] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:43.702059 15030 raft_consensus.cc:729] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45620bd03524564820f94f45f18b49f, State: Initialized, Role: FOLLOWER
I20260808 04:54:43.702255 15029 consensus_queue.cc:260] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.702529 15030 consensus_queue.cc:260] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.702819 15027 ts_tablet_manager.cc:1428] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:54:43.703118 15029 ts_tablet_manager.cc:1428] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:54:43.703147 14999 heartbeater.cc:503] Master 127.15.30.126:35217 was elected leader, sending a full tablet report...
I20260808 04:54:43.703476 15030 ts_tablet_manager.cc:1428] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 04:54:43.703470 14558 heartbeater.cc:503] Master 127.15.30.126:35217 was elected leader, sending a full tablet report...
I20260808 04:54:43.703780 14778 heartbeater.cc:503] Master 127.15.30.126:35217 was elected leader, sending a full tablet report...
W20260808 04:54:43.757267 14560 tablet.cc:2369] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:43.844553 15038 raft_consensus.cc:482] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:54:43.844789 15038 raft_consensus.cc:504] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.845966 15038 leader_election.cc:288] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a45620bd03524564820f94f45f18b49f (127.15.30.66:37389), 51604b193f214bf68dc8e14a473e65a9 (127.15.30.65:39803)
I20260808 04:54:43.851013 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.846468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:33147 (local address 127.15.30.66:37389)
0808 04:54:43.846627 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.846638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.846653 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:54:43.846703 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.846706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.846786 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.846880 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:43.847227 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.848028 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.849682 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.850275 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.850400 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.850450 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.850814 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.850883 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:54:43.851013 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.846673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.30.67:43913 (local address 127.15.30.65:39803)
0808 04:54:43.846825 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.846832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.847044 (+ 212us) server_negotiation.cc:407] Connection header received
0808 04:54:43.847125 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.847127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.847194 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.847281 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:54:43.847540 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.848224 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.849682 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.850284 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.850400 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.850450 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.850814 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.850883 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:54:43.851048 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.846220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.66:37389 (local address 127.15.30.67:33147)
0808 04:54:43.846360 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.846379 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.846492 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.846915 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.846918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.846942 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:43.847091 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.847102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.848144 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.848146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.849543 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.849556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.850418 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.850420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.850627 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.850703 (+ 76us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.850781 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.850881 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":223}
I20260808 04:54:43.851056 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.846555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:39803 (local address 127.15.30.67:43913)
0808 04:54:43.846928 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.846942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:54:43.847043 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:54:43.847303 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:54:43.847305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:54:43.847323 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:54:43.847424 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.847428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.848328 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.848331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.849557 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:54:43.849563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.850409 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.850413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:54:43.850602 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.850697 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 04:54:43.850788 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:54:43.850881 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":101,"thread_start_us":127,"threads_started":1}
I20260808 04:54:43.851795 14698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980119d8f7e4ec3ba7df0114ce174b0" candidate_uuid: "deec178cab0941a8bce243866a4cd77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45620bd03524564820f94f45f18b49f" is_pre_election: true
I20260808 04:54:43.851826 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980119d8f7e4ec3ba7df0114ce174b0" candidate_uuid: "deec178cab0941a8bce243866a4cd77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51604b193f214bf68dc8e14a473e65a9" is_pre_election: true
I20260808 04:54:43.852098 14698 raft_consensus.cc:2454] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deec178cab0941a8bce243866a4cd77d in term 0.
I20260808 04:54:43.852108 14475 raft_consensus.cc:2454] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deec178cab0941a8bce243866a4cd77d in term 0.
I20260808 04:54:43.852617 14812 leader_election.cc:302] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51604b193f214bf68dc8e14a473e65a9, deec178cab0941a8bce243866a4cd77d; no voters:
I20260808 04:54:43.852905 15038 raft_consensus.cc:2781] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:54:43.852985 15038 raft_consensus.cc:482] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:54:43.853014 15038 raft_consensus.cc:3037] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:43.854084 15038 raft_consensus.cc:504] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.854468 15038 leader_election.cc:288] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [CANDIDATE]: Term 1 election: Requested vote from peers a45620bd03524564820f94f45f18b49f (127.15.30.66:37389), 51604b193f214bf68dc8e14a473e65a9 (127.15.30.65:39803)
I20260808 04:54:43.854828 14698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980119d8f7e4ec3ba7df0114ce174b0" candidate_uuid: "deec178cab0941a8bce243866a4cd77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45620bd03524564820f94f45f18b49f"
I20260808 04:54:43.854915 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980119d8f7e4ec3ba7df0114ce174b0" candidate_uuid: "deec178cab0941a8bce243866a4cd77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51604b193f214bf68dc8e14a473e65a9"
I20260808 04:54:43.854983 14698 raft_consensus.cc:3037] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:43.855058 14475 raft_consensus.cc:3037] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:43.856086 14698 raft_consensus.cc:2454] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deec178cab0941a8bce243866a4cd77d in term 1.
I20260808 04:54:43.856205 14475 raft_consensus.cc:2454] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deec178cab0941a8bce243866a4cd77d in term 1.
I20260808 04:54:43.856480 14812 leader_election.cc:302] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a45620bd03524564820f94f45f18b49f, deec178cab0941a8bce243866a4cd77d; no voters:
I20260808 04:54:43.856648 15038 raft_consensus.cc:2781] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:43.856889 15038 raft_consensus.cc:686] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 1 LEADER]: Becoming Leader. State: Replica: deec178cab0941a8bce243866a4cd77d, State: Running, Role: LEADER
I20260808 04:54:43.857216 15038 consensus_queue.cc:237] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } }
I20260808 04:54:43.860575 14210 catalog_manager.cc:5495] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d reported cstate change: term changed from 0 to 1, leader changed from <none> to deec178cab0941a8bce243866a4cd77d (127.15.30.67). New cstate: current_term: 1 leader_uuid: "deec178cab0941a8bce243866a4cd77d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deec178cab0941a8bce243866a4cd77d" member_type: VOTER last_known_addr { host: "127.15.30.67" port: 36061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 } health_report { overall_health: UNKNOWN } } }
W20260808 04:54:43.882455 15000 tablet.cc:2369] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:54:43.912829 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.909215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.15.30.67:36061)
0808 04:54:43.909383 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:43.909389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:43.909405 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:54:43.909524 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:43.909527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:43.909592 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:43.909710 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:54:43.909962 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.910471 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.911331 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:43.911563 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:43.911673 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:43.911979 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:54:43.912400 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:54:43.912533 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:43.912605 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 04:54:43.912655 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:54:43.918741 14698 raft_consensus.cc:1261] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 1 FOLLOWER]: Refusing update from remote peer deec178cab0941a8bce243866a4cd77d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:43.918828 14475 raft_consensus.cc:1261] T a980119d8f7e4ec3ba7df0114ce174b0 P 51604b193f214bf68dc8e14a473e65a9 [term 1 FOLLOWER]: Refusing update from remote peer deec178cab0941a8bce243866a4cd77d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:54:43.919488 15043 consensus_queue.cc:1035] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51604b193f214bf68dc8e14a473e65a9" member_type: VOTER last_known_addr { host: "127.15.30.65" port: 39803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:43.919692 15038 consensus_queue.cc:1035] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a45620bd03524564820f94f45f18b49f" member_type: VOTER last_known_addr { host: "127.15.30.66" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:54:43.927528 15049 mvcc.cc:205] Tried to move back new op lower bound from 7316131364523544576 to 7316131364281774080. Current Snapshot: MvccSnapshot[applied={T|T < 7316131364523544576}]
I20260808 04:54:43.927696 15051 mvcc.cc:205] Tried to move back new op lower bound from 7316131364523544576 to 7316131364281774080. Current Snapshot: MvccSnapshot[applied={T|T < 7316131364523544576}]
I20260808 04:54:43.934801 14210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:43.934967 14210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:54:43.937559 14210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e796827141f4fe2aede99006f3866f0: Sending DeleteTablet for 3 replicas of tablet a980119d8f7e4ec3ba7df0114ce174b0
I20260808 04:54:43.938328 14884 tablet_service.cc:1514] Processing DeleteTablet for tablet a980119d8f7e4ec3ba7df0114ce174b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:43 UTC) from {username='slave'} at 127.0.0.1:58420
I20260808 04:54:43.938498 14432 tablet_service.cc:1514] Processing DeleteTablet for tablet a980119d8f7e4ec3ba7df0114ce174b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:43 UTC) from {username='slave'} at 127.0.0.1:38548
I20260808 04:54:43.938831 14665 tablet_service.cc:1514] Processing DeleteTablet for tablet a980119d8f7e4ec3ba7df0114ce174b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:54:43 UTC) from {username='slave'} at 127.0.0.1:37128
I20260808 04:54:43.939220 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 14328
I20260808 04:54:43.939299 15068 tablet_replica.cc:330] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: stopping tablet replica
I20260808 04:54:43.939611 15068 raft_consensus.cc:2229] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:54:43.939942 15068 raft_consensus.cc:2258] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:43.941872 15068 ts_tablet_manager.cc:1905] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:43.945942 15068 ts_tablet_manager.cc:1918] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:43.946094 15068 log.cc:1196] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-1/wal/wals/a980119d8f7e4ec3ba7df0114ce174b0
I20260808 04:54:43.946529 15068 ts_tablet_manager.cc:1939] T a980119d8f7e4ec3ba7df0114ce174b0 P a45620bd03524564820f94f45f18b49f: Deleting consensus metadata
I20260808 04:54:43.951061 14191 catalog_manager.cc:4841] TS a45620bd03524564820f94f45f18b49f (127.15.30.66:37389): tablet a980119d8f7e4ec3ba7df0114ce174b0 (table test-table [id=812fff17c9a74369913ac0649452ff16]) successfully deleted
I20260808 04:54:43.951301 15064 tablet_replica.cc:330] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: stopping tablet replica
I20260808 04:54:43.951655 15064 raft_consensus.cc:2229] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:54:43.952098 15064 raft_consensus.cc:2258] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:54:43.954208 15064 ts_tablet_manager.cc:1905] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:54:43.956311 15064 ts_tablet_manager.cc:1918] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:54:43.956459 15064 log.cc:1196] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Deleting WAL directory at /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786164620544947-15481-0/minicluster-data/ts-2/wal/wals/a980119d8f7e4ec3ba7df0114ce174b0
I20260808 04:54:43.956768 15064 ts_tablet_manager.cc:1939] T a980119d8f7e4ec3ba7df0114ce174b0 P deec178cab0941a8bce243866a4cd77d: Deleting consensus metadata
I20260808 04:54:43.957757 14191 catalog_manager.cc:4841] TS deec178cab0941a8bce243866a4cd77d (127.15.30.67:36061): tablet a980119d8f7e4ec3ba7df0114ce174b0 (table test-table [id=812fff17c9a74369913ac0649452ff16]) successfully deleted
W20260808 04:54:43.960546 14191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.30.65:39803 (error 108)
I20260808 04:54:43.961407 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 14562
I20260808 04:54:43.961598 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:43.961290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.65:39803 (local address 127.0.0.1:38562)
0808 04:54:43.961441 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:54:43.961513 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.65:39803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 04:54:43.974920 14191 catalog_manager.cc:4584] TS 51604b193f214bf68dc8e14a473e65a9 (127.15.30.65:39803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a980119d8f7e4ec3ba7df0114ce174b0: Network error: Client connection negotiation failed: client connection to 127.15.30.65:39803: connect: Connection refused (error 111)
I20260808 04:54:43.980358 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 14785
I20260808 04:54:43.992599 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 14159
2026-08-08T04:54:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1181 ms)
[----------] 6 tests from EncryptionPolicyTest (11336 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:44 dist-test-slave-zpvx krb5kdc[15073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:44 dist-test-slave-zpvx krb5kdc[15073](info): set up 2 sockets
Aug 08 04:54:44 dist-test-slave-zpvx krb5kdc[15073](info): commencing operation
krb5kdc: starting...
W20260808 04:54:45.246083 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:45 dist-test-slave-zpvx krb5kdc[15073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:45.367002 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:45.479218 15105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:45.479454 15105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:45.479494 15105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:45.483497 15105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:45.483549 15105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:45.483569 15105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:45.483585 15105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:45.483600 15105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:45.488732 15105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:45.489724 15105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:45.490710 15105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:45.499107 15112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:45.499118 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:45.499164 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:45.499200 15105 server_base.cc:1034] running on GCE node
I20260808 04:54:45.499919 15105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:45.501055 15105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:45.502236 15105 hybrid_clock.cc:648] HybridClock initialized: now 1786164885502221 us; error 27 us; skew 500 ppm
Aug 08 04:54:45 dist-test-slave-zpvx krb5kdc[15073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:45.505156 15105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:45.506026 15105 webserver.cc:460] Webserver started at http://127.0.0.1:40629/ using document root <none> and password file <none>
I20260808 04:54:45.506590 15105 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:45.506652 15105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:45.506901 15105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:45.509024 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "34db7049e224466e8645a3270a9070c4"
format_stamp: "Formatted at 2026-08-08 04:54:45 on dist-test-slave-zpvx"
server_key: "959b85762dee4ee31f2174d65bfa756d"
server_key_iv: "f46395a354bd8df92fa4ad8acb78fa59"
server_key_version: "encryptionkey@0"
I20260808 04:54:45.509624 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "34db7049e224466e8645a3270a9070c4"
format_stamp: "Formatted at 2026-08-08 04:54:45 on dist-test-slave-zpvx"
server_key: "959b85762dee4ee31f2174d65bfa756d"
server_key_iv: "f46395a354bd8df92fa4ad8acb78fa59"
server_key_version: "encryptionkey@0"
I20260808 04:54:45.513397 15105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:54:45.515899 15127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:45.517194 15105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:54:45.517361 15105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "34db7049e224466e8645a3270a9070c4"
format_stamp: "Formatted at 2026-08-08 04:54:45 on dist-test-slave-zpvx"
server_key: "959b85762dee4ee31f2174d65bfa756d"
server_key_iv: "f46395a354bd8df92fa4ad8acb78fa59"
server_key_version: "encryptionkey@0"
I20260808 04:54:45.517480 15105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:45.540510 15105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:45.543792 15105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:45.543992 15105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:45.551319 15105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38207
I20260808 04:54:45.551342 15214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38207 every 8 connection(s)
I20260808 04:54:45.552241 15105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:45.553208 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15105
I20260808 04:54:45.553402 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:45.553738 15481 external_mini_cluster.cc:1428] Setting key bfb1af5c07c464c9350b5efc71d05f47
I20260808 04:54:45.555644 15217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:45.561097 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4: Bootstrap starting.
Aug 08 04:54:45 dist-test-slave-zpvx krb5kdc[15073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:54:45.563710 15217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:45.564494 15217 log.cc:824] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:45.566694 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4: No bootstrap required, opened a new log
I20260808 04:54:45.568478 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:45.555175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46909 (local address 127.0.0.1:38207)
0808 04:54:45.555620 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:45.555635 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:45.555670 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:54:45.556355 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:45.556421 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:45.556772 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:45.557103 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:45.557935 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:45.558917 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:45.559810 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:45.560271 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:45.560468 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:45.563384 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:45.563389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:45.563401 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:45.563428 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:45.565453 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:45.566020 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:45.566024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:45.566028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:45.566137 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:45.566488 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:45.566490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:45.566491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:45.566929 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:45.567085 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:45.567436 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 04:54:45.567676 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":157,"threads_started":1}
I20260808 04:54:45.570925 15217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34db7049e224466e8645a3270a9070c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260808 04:54:45.571218 15217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:45.571295 15217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34db7049e224466e8645a3270a9070c4, State: Initialized, Role: FOLLOWER
I20260808 04:54:45.571746 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:45.571766 15217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34db7049e224466e8645a3270a9070c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260808 04:54:45.571967 15217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:45.572048 15217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:45.572149 15217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:45.573140 15217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34db7049e224466e8645a3270a9070c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260808 04:54:45.573503 15217 leader_election.cc:302] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34db7049e224466e8645a3270a9070c4; no voters:
I20260808 04:54:45.573854 15217 leader_election.cc:288] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:45.573925 15235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:45.574229 15235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [term 1 LEADER]: Becoming Leader. State: Replica: 34db7049e224466e8645a3270a9070c4, State: Running, Role: LEADER
I20260808 04:54:45.574554 15235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34db7049e224466e8645a3270a9070c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } }
I20260808 04:54:45.575850 15217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:45.576293 15243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34db7049e224466e8645a3270a9070c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34db7049e224466e8645a3270a9070c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } } }
I20260808 04:54:45.576452 15243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:45.576745 15241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34db7049e224466e8645a3270a9070c4. Latest consensus state: current_term: 1 leader_uuid: "34db7049e224466e8645a3270a9070c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34db7049e224466e8645a3270a9070c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38207 } } }
I20260808 04:54:45.576882 15241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4 [sys.catalog]: This master's current role is: LEADER
W20260808 04:54:45.581125 15266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:54:45.581255 15266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:54:45.581342 15246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:45.582201 15246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:45.583911 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:45.573974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45499 (local address 127.0.0.1:38207)
0808 04:54:45.574086 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:45.574090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:45.574469 (+ 379us) server_negotiation.cc:407] Connection header received
0808 04:54:45.574506 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:45.574509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:45.574551 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:45.574633 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:45.575325 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:45.576065 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:45.577550 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:45.577923 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:45.578452 (+ 529us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:45.579618 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:45.579621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:45.579623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:45.579644 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:45.581116 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:45.581660 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:45.581664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:45.581666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:45.581757 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:45.582100 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:45.582104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:45.582106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:45.582314 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:45.582457 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:45.583524 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0808 04:54:45.583682 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:54:45.588861 15246 catalog_manager.cc:1347] Generated new cluster ID: fdda1e2fd6ec44a1bcf1caa5d859da78
I20260808 04:54:45.588934 15246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:45.597699 15246 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:45.598522 15246 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:45.602965 15246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34db7049e224466e8645a3270a9070c4: Generated new TSK 0
I20260808 04:54:45.603549 15246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:45.612890 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15105
W20260808 04:54:45.625833 15070 connection.cc:535] client connection to 127.0.0.1:38207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38207): Connection reset by peer (error 104)
2026-08-08T04:54:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1622 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:45 dist-test-slave-zpvx krb5kdc[15271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:45 dist-test-slave-zpvx krb5kdc[15271](info): set up 2 sockets
Aug 08 04:54:45 dist-test-slave-zpvx krb5kdc[15271](info): commencing operation
krb5kdc: starting...
W20260808 04:54:46.860230 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:46 dist-test-slave-zpvx krb5kdc[15271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:46.985404 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:47.095486 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:47.095733 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:47.095773 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:47.099633 15295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:47.099687 15295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:47.099706 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:47.099722 15295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:47.099737 15295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:47.104780 15295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:47.105717 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:47.106657 15295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:47.115108 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:47.115296 15295 server_base.cc:1034] running on GCE node
W20260808 04:54:47.115069 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:47.115095 15303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:47.116086 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:47.117177 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:47.118436 15295 hybrid_clock.cc:648] HybridClock initialized: now 1786164887118415 us; error 32 us; skew 500 ppm
Aug 08 04:54:47 dist-test-slave-zpvx krb5kdc[15271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:47.121333 15295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:47.122196 15295 webserver.cc:460] Webserver started at http://127.0.0.1:33271/ using document root <none> and password file <none>
I20260808 04:54:47.122761 15295 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:47.122859 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:47.123095 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:47.125193 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "2f137c0d0649458aa42c0e9b0cff0e4e"
format_stamp: "Formatted at 2026-08-08 04:54:47 on dist-test-slave-zpvx"
server_key: "805eaef23b66288bdee5ade842138d76"
server_key_iv: "220ad15b69049850773997d098547d1c"
server_key_version: "encryptionkey@0"
I20260808 04:54:47.125779 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "2f137c0d0649458aa42c0e9b0cff0e4e"
format_stamp: "Formatted at 2026-08-08 04:54:47 on dist-test-slave-zpvx"
server_key: "805eaef23b66288bdee5ade842138d76"
server_key_iv: "220ad15b69049850773997d098547d1c"
server_key_version: "encryptionkey@0"
I20260808 04:54:47.129454 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:54:47.131820 15318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:47.133044 15295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:47.133196 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "2f137c0d0649458aa42c0e9b0cff0e4e"
format_stamp: "Formatted at 2026-08-08 04:54:47 on dist-test-slave-zpvx"
server_key: "805eaef23b66288bdee5ade842138d76"
server_key_iv: "220ad15b69049850773997d098547d1c"
server_key_version: "encryptionkey@0"
I20260808 04:54:47.133309 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:47.155515 15295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:47.158565 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:47.158787 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:47.166116 15295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34583
I20260808 04:54:47.166138 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34583 every 8 connection(s)
I20260808 04:54:47.167097 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:47.169858 15413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:47.171355 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15295
I20260808 04:54:47.171506 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:47.171831 15481 external_mini_cluster.cc:1428] Setting key aa7484d8114c02a1f4cf87c26839a75c
I20260808 04:54:47.176604 15413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e: Bootstrap starting.
I20260808 04:54:47.179443 15413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:47.180320 15413 log.cc:824] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:47.182034 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:47.173258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55925 (local address 127.0.0.1:34583)
0808 04:54:47.173684 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:47.173711 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:47.173769 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:54:47.174615 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:47.174638 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:47.175077 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:47.175444 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:47.175859 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:47.177407 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:47.178333 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:47.179005 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:47.179178 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:47.180117 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:47.180123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:47.180140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:47.180719 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:47.180792 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:47.180827 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:47.180977 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:47.181189 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 04:54:47.181329 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260808 04:54:47.182484 15413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e: No bootstrap required, opened a new log
I20260808 04:54:47.185181 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:47.185719 15413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260808 04:54:47.185963 15413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:47.186043 15413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f137c0d0649458aa42c0e9b0cff0e4e, State: Initialized, Role: FOLLOWER
I20260808 04:54:47.186465 15413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260808 04:54:47.186616 15413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:47.186691 15413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:47.186817 15413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:47.187960 15413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260808 04:54:47.188328 15413 leader_election.cc:302] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f137c0d0649458aa42c0e9b0cff0e4e; no voters:
I20260808 04:54:47.188700 15413 leader_election.cc:288] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:47.188827 15428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:47.189096 15428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [term 1 LEADER]: Becoming Leader. State: Replica: 2f137c0d0649458aa42c0e9b0cff0e4e, State: Running, Role: LEADER
I20260808 04:54:47.189472 15428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260808 04:54:47.189890 15413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:47.190742 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f137c0d0649458aa42c0e9b0cff0e4e. Latest consensus state: current_term: 1 leader_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } } }
I20260808 04:54:47.190948 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:47.191192 15429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f137c0d0649458aa42c0e9b0cff0e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } } }
I20260808 04:54:47.191310 15429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:47.191830 15441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:47.192329 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:47.187423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48319 (local address 127.0.0.1:34583)
0808 04:54:47.187522 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:47.187526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:47.187676 (+ 150us) server_negotiation.cc:407] Connection header received
0808 04:54:47.187803 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:47.187805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:47.187840 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:47.187902 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:47.188194 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:47.188879 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:47.190085 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:47.190418 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:47.190526 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:47.191498 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:47.191502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:47.191504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:47.191517 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:47.191543 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:47.191554 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:47.191646 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:47.192131 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:54:47.192169 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 04:54:47.194876 15441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:47.201356 15441 catalog_manager.cc:1347] Generated new cluster ID: e041fd83bcc542b8ad67278395453981
I20260808 04:54:47.201421 15441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:47.214184 15441 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:47.214995 15441 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:47.221104 15441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f137c0d0649458aa42c0e9b0cff0e4e: Generated new TSK 0
I20260808 04:54:47.221856 15441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:47.233523 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15295
W20260808 04:54:47.245064 15269 connection.cc:535] client connection to 127.0.0.1:34583 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34583): Connection reset by peer (error 104)
2026-08-08T04:54:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1619 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:47 dist-test-slave-zpvx krb5kdc[15469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:47 dist-test-slave-zpvx krb5kdc[15469](info): set up 2 sockets
Aug 08 04:54:47 dist-test-slave-zpvx krb5kdc[15469](info): commencing operation
krb5kdc: starting...
W20260808 04:54:48.479782 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:48 dist-test-slave-zpvx krb5kdc[15469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:48.602850 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:48.720983 15491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:48.721233 15491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:48.721273 15491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:48.725140 15491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:48.725199 15491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:48.725220 15491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:48.725237 15491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:48.725257 15491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:48.730297 15491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:48.731287 15491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:48.732296 15491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:48.740726 15497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:48.740738 15505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:48.740746 15499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:48.741006 15491 server_base.cc:1034] running on GCE node
I20260808 04:54:48.741535 15491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:48.742623 15491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:48.743816 15491 hybrid_clock.cc:648] HybridClock initialized: now 1786164888743786 us; error 42 us; skew 500 ppm
Aug 08 04:54:48 dist-test-slave-zpvx krb5kdc[15469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:48.752053 15491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:48.752988 15491 webserver.cc:460] Webserver started at http://127.0.0.1:44899/ using document root <none> and password file <none>
I20260808 04:54:48.753552 15491 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:48.753645 15491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:48.753871 15491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:48.755837 15491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "261f6cb5d162469aa39dacd825591114"
format_stamp: "Formatted at 2026-08-08 04:54:48 on dist-test-slave-zpvx"
server_key: "f48f824b811aac42f8491f325d86f770"
server_key_iv: "51941fecddf88c46b4cf1a6004140ce2"
server_key_version: "encryptionkey@0"
I20260808 04:54:48.756461 15491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "261f6cb5d162469aa39dacd825591114"
format_stamp: "Formatted at 2026-08-08 04:54:48 on dist-test-slave-zpvx"
server_key: "f48f824b811aac42f8491f325d86f770"
server_key_iv: "51941fecddf88c46b4cf1a6004140ce2"
server_key_version: "encryptionkey@0"
I20260808 04:54:48.760025 15491 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:54:48.762478 15515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:48.763665 15491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:48.763810 15491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "261f6cb5d162469aa39dacd825591114"
format_stamp: "Formatted at 2026-08-08 04:54:48 on dist-test-slave-zpvx"
server_key: "f48f824b811aac42f8491f325d86f770"
server_key_iv: "51941fecddf88c46b4cf1a6004140ce2"
server_key_version: "encryptionkey@0"
I20260808 04:54:48.763918 15491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:48.768327 15491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:48.771050 15491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:48.771252 15491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:48.778661 15491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37187
I20260808 04:54:48.778702 15616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37187 every 8 connection(s)
I20260808 04:54:48.779510 15491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:48.781292 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15491
I20260808 04:54:48.781446 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:48.781756 15481 external_mini_cluster.cc:1428] Setting key dea5a861ab308668d263351877acdd5a
I20260808 04:54:48.782501 15618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:48.786790 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:48.783109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39997 (local address 127.0.0.1:37187)
0808 04:54:48.783532 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:48.783547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:48.783580 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:54:48.784194 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:48.784241 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:48.784594 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:48.784921 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:48.785694 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:48.785698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:48.785709 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:48.785720 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:48.785769 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:48.785788 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:48.785935 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:48.786014 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 04:54:48.786159 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":142,"threads_started":1}
I20260808 04:54:48.788743 15618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114: Bootstrap starting.
I20260808 04:54:48.789633 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:48.791090 15618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:48.791877 15618 log.cc:824] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:48.793363 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:48.791694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42863 (local address 127.0.0.1:37187)
0808 04:54:48.791818 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:48.791824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:48.792073 (+ 249us) server_negotiation.cc:407] Connection header received
0808 04:54:48.792179 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:48.792182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:48.792210 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:48.792301 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:48.792981 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:48.792983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:48.792985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:48.792996 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:48.793020 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:48.793030 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:48.793105 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:48.793203 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 04:54:48.793230 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:54:48.794162 15618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114: No bootstrap required, opened a new log
I20260808 04:54:48.797648 15618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261f6cb5d162469aa39dacd825591114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260808 04:54:48.797932 15618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:48.797998 15618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261f6cb5d162469aa39dacd825591114, State: Initialized, Role: FOLLOWER
I20260808 04:54:48.798441 15618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261f6cb5d162469aa39dacd825591114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260808 04:54:48.798568 15618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:48.798655 15618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:48.798751 15618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:48.799652 15618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261f6cb5d162469aa39dacd825591114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260808 04:54:48.799986 15618 leader_election.cc:302] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261f6cb5d162469aa39dacd825591114; no voters:
I20260808 04:54:48.800287 15618 leader_election.cc:288] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:48.800434 15634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:48.800684 15634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [term 1 LEADER]: Becoming Leader. State: Replica: 261f6cb5d162469aa39dacd825591114, State: Running, Role: LEADER
I20260808 04:54:48.801034 15634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261f6cb5d162469aa39dacd825591114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } }
I20260808 04:54:48.801533 15618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:48.802736 15635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261f6cb5d162469aa39dacd825591114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261f6cb5d162469aa39dacd825591114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } } }
I20260808 04:54:48.802868 15635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:48.803277 15645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:48.803223 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261f6cb5d162469aa39dacd825591114. Latest consensus state: current_term: 1 leader_uuid: "261f6cb5d162469aa39dacd825591114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261f6cb5d162469aa39dacd825591114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37187 } } }
I20260808 04:54:48.803426 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:48.806270 15645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:48.811889 15645 catalog_manager.cc:1347] Generated new cluster ID: 19379926aa67470a8d76902c7dad66f9
I20260808 04:54:48.811957 15645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:48.835752 15645 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:48.836076 15645 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:48.843994 15645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261f6cb5d162469aa39dacd825591114: Generated new TSK 0
I20260808 04:54:48.844659 15645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:48.849712 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15491
2026-08-08T04:54:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1623 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:48 dist-test-slave-zpvx krb5kdc[15664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:48 dist-test-slave-zpvx krb5kdc[15664](info): set up 2 sockets
Aug 08 04:54:48 dist-test-slave-zpvx krb5kdc[15664](info): commencing operation
krb5kdc: starting...
W20260808 04:54:50.107372 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:50 dist-test-slave-zpvx krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:50.239735 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:50.350826 15685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:50.351078 15685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:50.351117 15685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:50.354935 15685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:50.354985 15685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:50.355003 15685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:50.355020 15685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:50.355036 15685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:50.360309 15685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:50.361258 15685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:50.362159 15685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:50.370407 15693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:50.370404 15699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:50.370493 15685 server_base.cc:1034] running on GCE node
W20260808 04:54:50.370443 15694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:50.371133 15685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:50.372169 15685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:50.373327 15685 hybrid_clock.cc:648] HybridClock initialized: now 1786164890373299 us; error 39 us; skew 500 ppm
Aug 08 04:54:50 dist-test-slave-zpvx krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:50.376008 15685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:50.377172 15685 webserver.cc:460] Webserver started at http://127.0.0.1:41933/ using document root <none> and password file <none>
I20260808 04:54:50.377805 15685 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:50.377920 15685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:50.378175 15685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:50.380323 15685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "68f58ac5563e4f129b6ef5b3aec88466"
format_stamp: "Formatted at 2026-08-08 04:54:50 on dist-test-slave-zpvx"
server_key: "3ff42240eca17890f15dff5960ad863a"
server_key_iv: "133ff77c6472f9bfe316430e7ce70c14"
server_key_version: "encryptionkey@0"
I20260808 04:54:50.381011 15685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "68f58ac5563e4f129b6ef5b3aec88466"
format_stamp: "Formatted at 2026-08-08 04:54:50 on dist-test-slave-zpvx"
server_key: "3ff42240eca17890f15dff5960ad863a"
server_key_iv: "133ff77c6472f9bfe316430e7ce70c14"
server_key_version: "encryptionkey@0"
I20260808 04:54:50.384938 15685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:54:50.387586 15709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:50.388954 15685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:50.389137 15685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "68f58ac5563e4f129b6ef5b3aec88466"
format_stamp: "Formatted at 2026-08-08 04:54:50 on dist-test-slave-zpvx"
server_key: "3ff42240eca17890f15dff5960ad863a"
server_key_iv: "133ff77c6472f9bfe316430e7ce70c14"
server_key_version: "encryptionkey@0"
I20260808 04:54:50.389257 15685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:50.402498 15685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:50.405773 15685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:50.405915 15685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:50.413043 15685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46143
I20260808 04:54:50.413125 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46143 every 8 connection(s)
I20260808 04:54:50.413964 15685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:50.415730 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15685
I20260808 04:54:50.415875 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:50.416172 15481 external_mini_cluster.cc:1428] Setting key 15de086ac68b52badb77d5734a87ac10
I20260808 04:54:50.417035 15801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:50.423815 15801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466: Bootstrap starting.
Aug 08 04:54:50 dist-test-slave-zpvx krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:54:50.426306 15801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:50.427294 15801 log.cc:824] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:50.429548 15801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466: No bootstrap required, opened a new log
I20260808 04:54:50.431130 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:50.417616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47655 (local address 127.0.0.1:46143)
0808 04:54:50.418040 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:50.418060 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:50.418103 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:54:50.419010 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:50.419033 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:50.419465 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:50.419829 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:50.420500 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:50.421984 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:50.422840 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:50.423506 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:50.423677 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:50.426082 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:50.426088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:50.426103 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:50.426137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:50.428018 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:50.428515 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:50.428521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:50.428527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:50.428667 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:50.428953 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:50.428956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:50.428958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:50.429462 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:50.429654 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:50.429983 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 04:54:50.430262 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260808 04:54:50.433858 15801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f58ac5563e4f129b6ef5b3aec88466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46143 } }
I20260808 04:54:50.434129 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:50.434131 15801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:50.434216 15801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f58ac5563e4f129b6ef5b3aec88466, State: Initialized, Role: FOLLOWER
I20260808 04:54:50.434639 15801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f58ac5563e4f129b6ef5b3aec88466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46143 } }
I20260808 04:54:50.434758 15801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:50.434852 15801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:50.434954 15801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:50.435941 15801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f58ac5563e4f129b6ef5b3aec88466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46143 } }
I20260808 04:54:50.436301 15801 leader_election.cc:302] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f58ac5563e4f129b6ef5b3aec88466; no voters:
I20260808 04:54:50.436702 15801 leader_election.cc:288] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:50.436812 15816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:50.437034 15816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [term 1 LEADER]: Becoming Leader. State: Replica: 68f58ac5563e4f129b6ef5b3aec88466, State: Running, Role: LEADER
I20260808 04:54:50.437420 15816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f58ac5563e4f129b6ef5b3aec88466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46143 } }
I20260808 04:54:50.437984 15801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:50.439006 15821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f58ac5563e4f129b6ef5b3aec88466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f58ac5563e4f129b6ef5b3aec88466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46143 } } }
I20260808 04:54:50.439141 15821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:50.439369 15821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f58ac5563e4f129b6ef5b3aec88466. Latest consensus state: current_term: 1 leader_uuid: "68f58ac5563e4f129b6ef5b3aec88466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f58ac5563e4f129b6ef5b3aec88466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46143 } } }
I20260808 04:54:50.439442 15821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:50.440529 15835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:50.443977 15835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:50.445163 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:50.436248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59417 (local address 127.0.0.1:46143)
0808 04:54:50.436400 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:50.436405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:50.436629 (+ 224us) server_negotiation.cc:407] Connection header received
0808 04:54:50.436769 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:50.436773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:50.436822 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:50.436906 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:50.437721 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:50.438489 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:50.439785 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:50.440123 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:50.440237 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:50.441952 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:50.441955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:50.441957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:50.441977 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:50.443446 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:50.443875 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:50.443879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:50.443881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:50.443961 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:50.444244 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:50.444249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:50.444251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:50.444499 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:50.444635 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:50.444887 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 04:54:50.445001 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:54:50.449486 15835 catalog_manager.cc:1347] Generated new cluster ID: 9b66ad7434e44361af9425fa568fcfa3
I20260808 04:54:50.449555 15835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:50.479815 15835 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:50.481124 15835 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:50.489446 15835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f58ac5563e4f129b6ef5b3aec88466: Generated new TSK 0
I20260808 04:54:50.490226 15835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:50.500419 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15685
2026-08-08T04:54:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1641 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:50 dist-test-slave-zpvx krb5kdc[15856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:50 dist-test-slave-zpvx krb5kdc[15856](info): set up 2 sockets
Aug 08 04:54:50 dist-test-slave-zpvx krb5kdc[15856](info): commencing operation
krb5kdc: starting...
W20260808 04:54:51.751322 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:51 dist-test-slave-zpvx krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:51.877790 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:51.988580 15880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:51.988824 15880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:51.988903 15880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:51.993083 15880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:51.993137 15880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:51.993157 15880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:51.993173 15880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:51.993224 15880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:51.998376 15880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:51.999362 15880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:52.000341 15880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:52.008956 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:52.008975 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:52.009087 15896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:52.008967 15880 server_base.cc:1034] running on GCE node
I20260808 04:54:52.009703 15880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:52.010807 15880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:52.011999 15880 hybrid_clock.cc:648] HybridClock initialized: now 1786164892011982 us; error 32 us; skew 500 ppm
Aug 08 04:54:52 dist-test-slave-zpvx krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:52.014789 15880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:52.015612 15880 webserver.cc:460] Webserver started at http://127.0.0.1:35341/ using document root <none> and password file <none>
I20260808 04:54:52.016149 15880 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:52.016242 15880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:52.016521 15880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:52.018525 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "fcfa543d49704b86bfff9343cf6782ac"
format_stamp: "Formatted at 2026-08-08 04:54:52 on dist-test-slave-zpvx"
server_key: "d2cdff52813c358db6d71b35c910b336"
server_key_iv: "61a3caec9245cf846659371034ad877e"
server_key_version: "encryptionkey@0"
I20260808 04:54:52.019119 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "fcfa543d49704b86bfff9343cf6782ac"
format_stamp: "Formatted at 2026-08-08 04:54:52 on dist-test-slave-zpvx"
server_key: "d2cdff52813c358db6d71b35c910b336"
server_key_iv: "61a3caec9245cf846659371034ad877e"
server_key_version: "encryptionkey@0"
I20260808 04:54:52.022862 15880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:54:52.025321 15907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:52.026484 15880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:52.026636 15880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "fcfa543d49704b86bfff9343cf6782ac"
format_stamp: "Formatted at 2026-08-08 04:54:52 on dist-test-slave-zpvx"
server_key: "d2cdff52813c358db6d71b35c910b336"
server_key_iv: "61a3caec9245cf846659371034ad877e"
server_key_version: "encryptionkey@0"
I20260808 04:54:52.026746 15880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:52.038712 15880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:52.041440 15880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:52.041646 15880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:52.048553 15880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35097
I20260808 04:54:52.048601 16004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35097 every 8 connection(s)
I20260808 04:54:52.049501 15880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:52.052251 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:52.053857 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 15880
I20260808 04:54:52.054029 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:52.054333 15481 external_mini_cluster.cc:1428] Setting key f8e7d578ab161fa79cfd311fe33a991c
I20260808 04:54:52.059020 16005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac: Bootstrap starting.
I20260808 04:54:52.061550 16005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:52.062480 16005 log.cc:824] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:52.064189 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:52.055756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50815 (local address 127.0.0.1:35097)
0808 04:54:52.056168 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:52.056188 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:52.056230 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:54:52.057119 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:52.057144 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:52.057555 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:52.057925 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:52.058314 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:52.059738 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:52.060742 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:52.061395 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:52.061562 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:52.062473 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:52.062480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:52.062497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:52.062995 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:52.063046 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:52.063069 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:52.063182 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:52.063343 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 04:54:52.063536 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":104,"threads_started":1}
I20260808 04:54:52.064572 16005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac: No bootstrap required, opened a new log
I20260808 04:54:52.066875 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:52.068233 16005 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfa543d49704b86bfff9343cf6782ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35097 } }
I20260808 04:54:52.068518 16005 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:52.068608 16005 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfa543d49704b86bfff9343cf6782ac, State: Initialized, Role: FOLLOWER
I20260808 04:54:52.069106 16005 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfa543d49704b86bfff9343cf6782ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35097 } }
I20260808 04:54:52.069286 16005 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:52.069365 16005 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:52.069485 16005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:52.070501 16005 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfa543d49704b86bfff9343cf6782ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35097 } }
I20260808 04:54:52.070866 16005 leader_election.cc:302] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfa543d49704b86bfff9343cf6782ac; no voters:
I20260808 04:54:52.071193 16005 leader_election.cc:288] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:52.071306 16023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:52.071550 16023 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [term 1 LEADER]: Becoming Leader. State: Replica: fcfa543d49704b86bfff9343cf6782ac, State: Running, Role: LEADER
I20260808 04:54:52.071939 16023 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfa543d49704b86bfff9343cf6782ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35097 } }
I20260808 04:54:52.072484 16005 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:52.073318 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:52.068991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52533 (local address 127.0.0.1:35097)
0808 04:54:52.069124 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:52.069128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:52.069310 (+ 182us) server_negotiation.cc:407] Connection header received
0808 04:54:52.069467 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:52.069470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:52.069510 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:52.069616 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:52.069921 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:52.070429 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:52.071579 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:52.071899 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:52.072008 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:52.072828 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:52.072832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:52.072835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:52.072847 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:52.072881 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:52.072893 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:52.072969 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:52.073110 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 04:54:52.073160 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:54:52.073613 16023 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfa543d49704b86bfff9343cf6782ac. Latest consensus state: current_term: 1 leader_uuid: "fcfa543d49704b86bfff9343cf6782ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfa543d49704b86bfff9343cf6782ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35097 } } }
I20260808 04:54:52.073732 16023 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:52.074184 16024 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcfa543d49704b86bfff9343cf6782ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfa543d49704b86bfff9343cf6782ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35097 } } }
I20260808 04:54:52.074536 16024 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:52.074615 16043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:52.077644 16043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:52.082914 16043 catalog_manager.cc:1347] Generated new cluster ID: 91d94eee1f6647489109eca2607b4cfb
I20260808 04:54:52.082980 16043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:52.094895 16043 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:52.095758 16043 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:52.101853 16043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcfa543d49704b86bfff9343cf6782ac: Generated new TSK 0
I20260808 04:54:52.102517 16043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:52.116935 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 15880
2026-08-08T04:54:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1619 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:52 dist-test-slave-zpvx krb5kdc[16064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:52 dist-test-slave-zpvx krb5kdc[16064](info): set up 2 sockets
Aug 08 04:54:52 dist-test-slave-zpvx krb5kdc[16064](info): commencing operation
krb5kdc: starting...
W20260808 04:54:53.381557 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:53 dist-test-slave-zpvx krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:53.505206 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:53.614018 16088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:53.614252 16088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:53.614333 16088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:53.619664 16088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:53.619722 16088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:53.619747 16088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:53.619764 16088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:53.619779 16088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:53.625094 16088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:53.626025 16088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:53.626948 16088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:53.635318 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:53.635330 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:53.635330 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:53.635560 16088 server_base.cc:1034] running on GCE node
I20260808 04:54:53.636233 16088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:53.637287 16088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:53.638471 16088 hybrid_clock.cc:648] HybridClock initialized: now 1786164893638446 us; error 42 us; skew 500 ppm
Aug 08 04:54:53 dist-test-slave-zpvx krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:53.641286 16088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:53.642161 16088 webserver.cc:460] Webserver started at http://127.0.0.1:45907/ using document root <none> and password file <none>
I20260808 04:54:53.642697 16088 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:53.642802 16088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:53.643033 16088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:53.645144 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "06cd34d69ac44d5f986aa057d305cd34"
format_stamp: "Formatted at 2026-08-08 04:54:53 on dist-test-slave-zpvx"
server_key: "e8176079024f157fc53c532fbe71edc1"
server_key_iv: "381221eab699a9ba33626e5789776985"
server_key_version: "encryptionkey@0"
I20260808 04:54:53.645692 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "06cd34d69ac44d5f986aa057d305cd34"
format_stamp: "Formatted at 2026-08-08 04:54:53 on dist-test-slave-zpvx"
server_key: "e8176079024f157fc53c532fbe71edc1"
server_key_iv: "381221eab699a9ba33626e5789776985"
server_key_version: "encryptionkey@0"
I20260808 04:54:53.649243 16088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:54:53.651573 16112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:53.652808 16088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:53.652945 16088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "06cd34d69ac44d5f986aa057d305cd34"
format_stamp: "Formatted at 2026-08-08 04:54:53 on dist-test-slave-zpvx"
server_key: "e8176079024f157fc53c532fbe71edc1"
server_key_iv: "381221eab699a9ba33626e5789776985"
server_key_version: "encryptionkey@0"
I20260808 04:54:53.653051 16088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:53.657509 16088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:53.660326 16088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:53.660651 16088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:53.667953 16088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42837
I20260808 04:54:53.667965 16205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42837 every 8 connection(s)
I20260808 04:54:53.668933 16088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:53.670886 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16088
I20260808 04:54:53.671065 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:53.671380 15481 external_mini_cluster.cc:1428] Setting key c23d4a5328653f55ef167905945bc7eb
I20260808 04:54:53.672016 16206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:53.676575 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:53.672780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43761 (local address 127.0.0.1:42837)
0808 04:54:53.673167 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:53.673192 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:53.673227 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:54:53.673877 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:53.673895 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:53.674164 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:53.674437 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:53.675314 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:53.675319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:53.675330 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:53.675341 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:53.675395 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:53.675413 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:53.675524 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:53.675663 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 04:54:53.675853 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":126,"threads_started":1}
I20260808 04:54:53.678697 16206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34: Bootstrap starting.
I20260808 04:54:53.679423 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:53.681162 16206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:53.682123 16206 log.cc:824] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:53.683034 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:53.681422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57315 (local address 127.0.0.1:42837)
0808 04:54:53.681548 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:53.681553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:53.681725 (+ 172us) server_negotiation.cc:407] Connection header received
0808 04:54:53.681834 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:53.681836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:53.681863 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:53.681925 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:53.682672 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:53.682675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:53.682678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:53.682690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:53.682718 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:53.682734 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:53.682850 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:53.682888 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 04:54:53.682925 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:54:53.684420 16206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34: No bootstrap required, opened a new log
I20260808 04:54:53.687853 16206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd34d69ac44d5f986aa057d305cd34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260808 04:54:53.688087 16206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:53.688164 16206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cd34d69ac44d5f986aa057d305cd34, State: Initialized, Role: FOLLOWER
I20260808 04:54:53.688616 16206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd34d69ac44d5f986aa057d305cd34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260808 04:54:53.688774 16206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:53.688843 16206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:53.688939 16206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:53.689890 16206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd34d69ac44d5f986aa057d305cd34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260808 04:54:53.690219 16206 leader_election.cc:302] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cd34d69ac44d5f986aa057d305cd34; no voters:
I20260808 04:54:53.690513 16206 leader_election.cc:288] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:53.690646 16224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:53.690882 16224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [term 1 LEADER]: Becoming Leader. State: Replica: 06cd34d69ac44d5f986aa057d305cd34, State: Running, Role: LEADER
I20260808 04:54:53.691220 16224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd34d69ac44d5f986aa057d305cd34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260808 04:54:53.691704 16206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:53.692905 16229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cd34d69ac44d5f986aa057d305cd34. Latest consensus state: current_term: 1 leader_uuid: "06cd34d69ac44d5f986aa057d305cd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd34d69ac44d5f986aa057d305cd34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } } }
I20260808 04:54:53.692907 16227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cd34d69ac44d5f986aa057d305cd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd34d69ac44d5f986aa057d305cd34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } } }
I20260808 04:54:53.693054 16227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:53.693054 16229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:53.693496 16239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:53.696342 16239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:53.701529 16239 catalog_manager.cc:1347] Generated new cluster ID: f2c9a0041e7540688a73d8328651ed40
I20260808 04:54:53.701589 16239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:53.708470 16239 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:53.708737 16239 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:53.712430 16239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06cd34d69ac44d5f986aa057d305cd34: Generated new TSK 0
I20260808 04:54:53.713133 16239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:53.722285 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16088
2026-08-08T04:54:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1608 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:53 dist-test-slave-zpvx krb5kdc[16260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:53 dist-test-slave-zpvx krb5kdc[16260](info): set up 2 sockets
Aug 08 04:54:53 dist-test-slave-zpvx krb5kdc[16260](info): commencing operation
krb5kdc: starting...
W20260808 04:54:54.980000 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:55 dist-test-slave-zpvx krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:55.103720 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:55.213603 16279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:55.213897 16279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:55.213976 16279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:55.218111 16279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:55.218194 16279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:55.218230 16279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:55.218268 16279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:55.218302 16279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:55.223649 16279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:55.224813 16279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:55.225844 16279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:55.234491 16287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:55.234503 16286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:55.234512 16289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:55.234644 16279 server_base.cc:1034] running on GCE node
I20260808 04:54:55.235328 16279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:55.236461 16279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:55.237677 16279 hybrid_clock.cc:648] HybridClock initialized: now 1786164895237664 us; error 28 us; skew 500 ppm
Aug 08 04:54:55 dist-test-slave-zpvx krb5kdc[16260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:55.240502 16279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:55.241326 16279 webserver.cc:460] Webserver started at http://127.0.0.1:41409/ using document root <none> and password file <none>
I20260808 04:54:55.241853 16279 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:55.241910 16279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:55.242070 16279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:55.244097 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "3af6f19a545d4994878a729ed5238576"
format_stamp: "Formatted at 2026-08-08 04:54:55 on dist-test-slave-zpvx"
server_key: "77a06ebc0bea84492f3c2a50b25bb531"
server_key_iv: "b5c721c869fa4aab58685b4ad26cc359"
server_key_version: "encryptionkey@0"
I20260808 04:54:55.244709 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "3af6f19a545d4994878a729ed5238576"
format_stamp: "Formatted at 2026-08-08 04:54:55 on dist-test-slave-zpvx"
server_key: "77a06ebc0bea84492f3c2a50b25bb531"
server_key_iv: "b5c721c869fa4aab58685b4ad26cc359"
server_key_version: "encryptionkey@0"
I20260808 04:54:55.248363 16279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:54:55.250881 16301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:55.252122 16279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:55.252233 16279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "3af6f19a545d4994878a729ed5238576"
format_stamp: "Formatted at 2026-08-08 04:54:55 on dist-test-slave-zpvx"
server_key: "77a06ebc0bea84492f3c2a50b25bb531"
server_key_iv: "b5c721c869fa4aab58685b4ad26cc359"
server_key_version: "encryptionkey@0"
I20260808 04:54:55.252321 16279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:55.272861 16279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:55.275877 16279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:55.276033 16279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:55.283408 16279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36941
I20260808 04:54:55.283432 16389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36941 every 8 connection(s)
I20260808 04:54:55.284325 16279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:55.287070 16390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:55.289824 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16279
I20260808 04:54:55.289969 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:55.290270 15481 external_mini_cluster.cc:1428] Setting key 5d8a449621c0ae630516007a98719f1b
I20260808 04:54:55.293196 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576: Bootstrap starting.
I20260808 04:54:55.295372 16390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:55.296164 16390 log.cc:824] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:55.298250 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576: No bootstrap required, opened a new log
Aug 08 04:54:55 dist-test-slave-zpvx krb5kdc[16260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:54:55.301774 16390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af6f19a545d4994878a729ed5238576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36941 } }
I20260808 04:54:55.302016 16390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:55.302100 16390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af6f19a545d4994878a729ed5238576, State: Initialized, Role: FOLLOWER
I20260808 04:54:55.302583 16390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af6f19a545d4994878a729ed5238576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36941 } }
I20260808 04:54:55.302739 16390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:55.302826 16390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:55.302927 16390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:55.304072 16390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af6f19a545d4994878a729ed5238576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36941 } }
I20260808 04:54:55.304458 16390 leader_election.cc:302] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af6f19a545d4994878a729ed5238576; no voters:
I20260808 04:54:55.304804 16390 leader_election.cc:288] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:55.304934 16405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:55.305177 16405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [term 1 LEADER]: Becoming Leader. State: Replica: 3af6f19a545d4994878a729ed5238576, State: Running, Role: LEADER
I20260808 04:54:55.305467 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:55.291713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:50071 (local address 127.0.0.1:36941)
0808 04:54:55.292125 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:55.292144 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:55.292178 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:54:55.292836 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:55.292855 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:55.293181 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:55.293541 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:55.294360 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:55.295341 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:55.296215 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:55.296955 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:55.297111 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:55.299816 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:55.299824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:55.299842 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:55.299876 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:55.302082 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:55.302588 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:55.302594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:55.302602 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:55.302716 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:55.303142 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:55.303149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:55.303154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:55.303663 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:55.303902 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:55.304151 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 04:54:55.304445 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260808 04:54:55.305493 16405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af6f19a545d4994878a729ed5238576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36941 } }
I20260808 04:54:55.306061 16390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:55.306638 16405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af6f19a545d4994878a729ed5238576. Latest consensus state: current_term: 1 leader_uuid: "3af6f19a545d4994878a729ed5238576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af6f19a545d4994878a729ed5238576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36941 } } }
I20260808 04:54:55.306756 16405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:55.307397 16416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:55.307365 16411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af6f19a545d4994878a729ed5238576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af6f19a545d4994878a729ed5238576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36941 } } }
I20260808 04:54:55.307466 16411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:55.308781 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:55.311474 16416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:55.317756 16416 catalog_manager.cc:1347] Generated new cluster ID: 6eaf6030dcae4a48810998f79739e5f0
I20260808 04:54:55.317821 16416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:55.320202 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:55.310854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:34981 (local address 127.0.0.1:36941)
0808 04:54:55.310949 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:55.310955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:55.311355 (+ 400us) server_negotiation.cc:407] Connection header received
0808 04:54:55.311476 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:55.311479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:55.311519 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:55.311588 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:55.312696 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:55.313212 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:55.314319 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:55.314686 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:55.314805 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:55.316606 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:55.316610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:55.316613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:54:55.316636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:55.318336 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:55.318892 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:55.318896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:55.318897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:55.318971 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:54:55.319322 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:54:55.319326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:54:55.319328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:54:55.319500 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:55.319620 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:55.319977 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 04:54:55.320067 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 04:54:55.328094 16416 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:55.329074 16416 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:55.334153 16416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af6f19a545d4994878a729ed5238576: Generated new TSK 0
I20260808 04:54:55.334733 16416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:55.339596 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16279
W20260808 04:54:55.353911 16258 connection.cc:535] client connection to 127.0.0.1:36941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36941): Connection reset by peer (error 104)
2026-08-08T04:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1615 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:55 dist-test-slave-zpvx krb5kdc[16447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:55 dist-test-slave-zpvx krb5kdc[16447](info): set up 2 sockets
Aug 08 04:54:55 dist-test-slave-zpvx krb5kdc[16447](info): commencing operation
krb5kdc: starting...
W20260808 04:54:56.591568 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:56 dist-test-slave-zpvx krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:56.721094 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:56.831630 16467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:56.831915 16467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:56.831987 16467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:56.835889 16467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:56.835974 16467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:56.836011 16467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:56.836069 16467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:56.836109 16467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:56.841195 16467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:56.842164 16467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:56.843165 16467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:56.851562 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:56.851543 16481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:56.851585 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:56.851722 16467 server_base.cc:1034] running on GCE node
I20260808 04:54:56.852481 16467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:56.853552 16467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:56.854744 16467 hybrid_clock.cc:648] HybridClock initialized: now 1786164896854729 us; error 27 us; skew 500 ppm
Aug 08 04:54:56 dist-test-slave-zpvx krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:56.857481 16467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:56.858315 16467 webserver.cc:460] Webserver started at http://127.0.0.1:36815/ using document root <none> and password file <none>
I20260808 04:54:56.858856 16467 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:56.858945 16467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:56.859187 16467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:56.861340 16467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "c2b52f81557349fc95ab3eead41ecab5"
format_stamp: "Formatted at 2026-08-08 04:54:56 on dist-test-slave-zpvx"
server_key: "6544ce284e3d3a7a0f01aa878f28a036"
server_key_iv: "d4bffcb64559a35ad7d7093786c9d18c"
server_key_version: "encryptionkey@0"
I20260808 04:54:56.861912 16467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "c2b52f81557349fc95ab3eead41ecab5"
format_stamp: "Formatted at 2026-08-08 04:54:56 on dist-test-slave-zpvx"
server_key: "6544ce284e3d3a7a0f01aa878f28a036"
server_key_iv: "d4bffcb64559a35ad7d7093786c9d18c"
server_key_version: "encryptionkey@0"
I20260808 04:54:56.865644 16467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:54:56.868057 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:56.869386 16467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:54:56.869526 16467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "c2b52f81557349fc95ab3eead41ecab5"
format_stamp: "Formatted at 2026-08-08 04:54:56 on dist-test-slave-zpvx"
server_key: "6544ce284e3d3a7a0f01aa878f28a036"
server_key_iv: "d4bffcb64559a35ad7d7093786c9d18c"
server_key_version: "encryptionkey@0"
I20260808 04:54:56.869635 16467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:56.883647 16467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:56.886663 16467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:56.886875 16467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:56.894467 16467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37685
I20260808 04:54:56.894495 16587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37685 every 8 connection(s)
I20260808 04:54:56.895417 16467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:56.896956 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16467
I20260808 04:54:56.897113 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:56.897437 15481 external_mini_cluster.cc:1428] Setting key 4f6ee40264171050252b80ada5028a1c
I20260808 04:54:56.898365 16588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:56.904156 16588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5: Bootstrap starting.
I20260808 04:54:56.906308 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:56.898842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:45471 (local address 127.0.0.1:37685)
0808 04:54:56.899296 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:56.899310 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:56.899351 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:54:56.899977 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:56.899993 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:56.900316 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:56.900713 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:56.901110 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:56.902085 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:56.902985 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:56.903603 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:56.903794 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:56.904729 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:56.904734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:56.904746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:56.905109 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:56.905163 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:56.905185 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:56.905330 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:56.905485 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 04:54:56.905621 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":159,"threads_started":1}
I20260808 04:54:56.906450 16588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:56.907182 16588 log.cc:824] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:56.909392 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:56.909524 16588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5: No bootstrap required, opened a new log
I20260808 04:54:56.913642 16588 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b52f81557349fc95ab3eead41ecab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37685 } }
I20260808 04:54:56.913882 16588 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:56.913944 16588 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2b52f81557349fc95ab3eead41ecab5, State: Initialized, Role: FOLLOWER
I20260808 04:54:56.914403 16588 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b52f81557349fc95ab3eead41ecab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37685 } }
I20260808 04:54:56.914528 16588 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:56.914619 16588 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:56.914736 16588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:56.915602 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:56.911574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:36755 (local address 127.0.0.1:37685)
0808 04:54:56.911686 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:56.911690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:56.911912 (+ 222us) server_negotiation.cc:407] Connection header received
0808 04:54:56.912063 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:56.912066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:56.912113 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:56.912221 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:56.912534 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:56.913042 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:56.913886 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:54:56.914207 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:54:56.914318 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:54:56.915154 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:56.915158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:56.915161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:56.915174 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:56.915204 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:56.915214 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:56.915313 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:56.915435 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:54:56.915466 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:54:56.915688 16588 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b52f81557349fc95ab3eead41ecab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37685 } }
I20260808 04:54:56.916051 16588 leader_election.cc:302] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2b52f81557349fc95ab3eead41ecab5; no voters:
I20260808 04:54:56.916417 16588 leader_election.cc:288] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:56.916489 16605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:56.916697 16605 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [term 1 LEADER]: Becoming Leader. State: Replica: c2b52f81557349fc95ab3eead41ecab5, State: Running, Role: LEADER
I20260808 04:54:56.917032 16605 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b52f81557349fc95ab3eead41ecab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37685 } }
I20260808 04:54:56.917654 16588 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:56.918226 16605 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2b52f81557349fc95ab3eead41ecab5. Latest consensus state: current_term: 1 leader_uuid: "c2b52f81557349fc95ab3eead41ecab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b52f81557349fc95ab3eead41ecab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37685 } } }
I20260808 04:54:56.918424 16605 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:56.918645 16606 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2b52f81557349fc95ab3eead41ecab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b52f81557349fc95ab3eead41ecab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37685 } } }
I20260808 04:54:56.918797 16606 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:56.918917 16620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:56.922079 16620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:56.927752 16620 catalog_manager.cc:1347] Generated new cluster ID: f9b44841c0d048f9afca08a3f9c1c701
I20260808 04:54:56.927829 16620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:56.944185 16620 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:56.945901 16620 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:56.953681 16620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2b52f81557349fc95ab3eead41ecab5: Generated new TSK 0
I20260808 04:54:56.954347 16620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:56.960192 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16467
W20260808 04:54:56.973186 16445 connection.cc:535] client connection to 127.0.0.1:37685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37685): Connection reset by peer (error 104)
2026-08-08T04:54:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1619 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:57 dist-test-slave-zpvx krb5kdc[16639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:57 dist-test-slave-zpvx krb5kdc[16639](info): set up 2 sockets
Aug 08 04:54:57 dist-test-slave-zpvx krb5kdc[16639](info): commencing operation
krb5kdc: starting...
W20260808 04:54:58.232487 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:58 dist-test-slave-zpvx krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:58.357465 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:54:58.473919 16664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:54:58.474215 16664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:54:58.474299 16664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:54:58.478245 16664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:54:58.478330 16664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:54:58.478368 16664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:54:58.478425 16664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:54:58.478466 16664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:54:58.483471 16664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:54:58.484491 16664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:54:58.485482 16664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:54:58.493837 16673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:58.493858 16677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:54:58.493845 16675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:54:58.494066 16664 server_base.cc:1034] running on GCE node
I20260808 04:54:58.494683 16664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:54:58.495703 16664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:54:58.496897 16664 hybrid_clock.cc:648] HybridClock initialized: now 1786164898496878 us; error 32 us; skew 500 ppm
Aug 08 04:54:58 dist-test-slave-zpvx krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:54:58.499590 16664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:54:58.500552 16664 webserver.cc:460] Webserver started at http://127.0.0.1:46669/ using document root <none> and password file <none>
I20260808 04:54:58.501093 16664 fs_manager.cc:359] Metadata directory not provided
I20260808 04:54:58.501184 16664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:54:58.501412 16664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:54:58.503472 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "62fd57408e62442dba42f6b920a62f8f"
format_stamp: "Formatted at 2026-08-08 04:54:58 on dist-test-slave-zpvx"
server_key: "3c0e37eea6b25c86cf187819bc8ab0bf"
server_key_iv: "25d7801e69d694aa6e71f1a9f77f8fb7"
server_key_version: "encryptionkey@0"
I20260808 04:54:58.504045 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "62fd57408e62442dba42f6b920a62f8f"
format_stamp: "Formatted at 2026-08-08 04:54:58 on dist-test-slave-zpvx"
server_key: "3c0e37eea6b25c86cf187819bc8ab0bf"
server_key_iv: "25d7801e69d694aa6e71f1a9f77f8fb7"
server_key_version: "encryptionkey@0"
I20260808 04:54:58.507943 16664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:54:58.510437 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:54:58.511740 16664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:54:58.511894 16664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "62fd57408e62442dba42f6b920a62f8f"
format_stamp: "Formatted at 2026-08-08 04:54:58 on dist-test-slave-zpvx"
server_key: "3c0e37eea6b25c86cf187819bc8ab0bf"
server_key_iv: "25d7801e69d694aa6e71f1a9f77f8fb7"
server_key_version: "encryptionkey@0"
I20260808 04:54:58.512006 16664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:54:58.516615 16664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:54:58.519503 16664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:54:58.519702 16664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:54:58.526980 16664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34883
I20260808 04:54:58.527009 16797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34883 every 8 connection(s)
I20260808 04:54:58.527875 16664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:54:58.530584 16801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:54:58.533560 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16664
I20260808 04:54:58.533715 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:54:58.534017 15481 external_mini_cluster.cc:1428] Setting key 16241dc48c9876ace532523396a09a95
I20260808 04:54:58.536628 16801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f: Bootstrap starting.
I20260808 04:54:58.539405 16801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f: Neither blocks nor log segments found. Creating new log.
I20260808 04:54:58.539685 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:58.535479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:38089 (local address 127.0.0.1:34883)
0808 04:54:58.535916 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:58.535933 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:58.535975 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:54:58.536822 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:58.536846 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:58.537191 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:58.537560 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:58.538316 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:58.538322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:58.538336 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:58.538349 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:58.538411 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:58.538436 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:58.538576 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:58.538641 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 04:54:58.538788 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":109,"threads_started":1}
I20260808 04:54:58.540267 16801 log.cc:824] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:54:58.542289 16801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f: No bootstrap required, opened a new log
I20260808 04:54:58.542577 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:54:58.546633 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:54:58.544891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:55945 (local address 127.0.0.1:34883)
0808 04:54:58.545018 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:54:58.545022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:54:58.545249 (+ 227us) server_negotiation.cc:407] Connection header received
0808 04:54:58.545341 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:54:58.545343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:54:58.545373 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:54:58.545440 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:54:58.546187 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:54:58.546190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:54:58.546191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:54:58.546202 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:54:58.546228 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:54:58.546238 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:54:58.546309 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:54:58.546424 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 04:54:58.546466 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:54:58.547168 16801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fd57408e62442dba42f6b920a62f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260808 04:54:58.547379 16801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:54:58.547437 16801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fd57408e62442dba42f6b920a62f8f, State: Initialized, Role: FOLLOWER
I20260808 04:54:58.547894 16801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fd57408e62442dba42f6b920a62f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260808 04:54:58.548018 16801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:54:58.548110 16801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:54:58.548218 16801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:54:58.549166 16801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fd57408e62442dba42f6b920a62f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260808 04:54:58.549506 16801 leader_election.cc:302] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62fd57408e62442dba42f6b920a62f8f; no voters:
I20260808 04:54:58.549831 16801 leader_election.cc:288] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:54:58.549958 16812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:54:58.550213 16812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [term 1 LEADER]: Becoming Leader. State: Replica: 62fd57408e62442dba42f6b920a62f8f, State: Running, Role: LEADER
I20260808 04:54:58.550552 16812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fd57408e62442dba42f6b920a62f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260808 04:54:58.551065 16801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:54:58.552230 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62fd57408e62442dba42f6b920a62f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fd57408e62442dba42f6b920a62f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } } }
I20260808 04:54:58.552361 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:58.552716 16815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62fd57408e62442dba42f6b920a62f8f. Latest consensus state: current_term: 1 leader_uuid: "62fd57408e62442dba42f6b920a62f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62fd57408e62442dba42f6b920a62f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } } }
I20260808 04:54:58.552806 16826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:54:58.552876 16815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f [sys.catalog]: This master's current role is: LEADER
I20260808 04:54:58.556250 16826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:54:58.561749 16826 catalog_manager.cc:1347] Generated new cluster ID: 93e05f84b009486e9929001721c50c50
I20260808 04:54:58.561837 16826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:54:58.579722 16826 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:54:58.580037 16826 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:54:58.585985 16826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62fd57408e62442dba42f6b920a62f8f: Generated new TSK 0
I20260808 04:54:58.586632 16826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:54:58.596632 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16664
2026-08-08T04:54:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1634 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:54:58 dist-test-slave-zpvx krb5kdc[16848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:54:58 dist-test-slave-zpvx krb5kdc[16848](info): set up 2 sockets
Aug 08 04:54:58 dist-test-slave-zpvx krb5kdc[16848](info): commencing operation
krb5kdc: starting...
W20260808 04:54:59.853796 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:54:59 dist-test-slave-zpvx krb5kdc[16848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:54:59.980818 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:00.095851 16868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:00.096164 16868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:00.096241 16868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:00.100194 16868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:00.100297 16868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:00.100337 16868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:00.100411 16868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:00.100466 16868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:00.105715 16868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:00.106750 16868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:00.107790 16868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:00.116314 16876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:00.116421 16875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:00.116648 16868 server_base.cc:1034] running on GCE node
W20260808 04:55:00.116309 16878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:00.117260 16868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:00.118360 16868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:00.119577 16868 hybrid_clock.cc:648] HybridClock initialized: now 1786164900119549 us; error 39 us; skew 500 ppm
Aug 08 04:55:00 dist-test-slave-zpvx krb5kdc[16848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:00.126689 16868 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:55:00.127591 16868 webserver.cc:460] Webserver started at http://127.0.0.1:38449/ using document root <none> and password file <none>
I20260808 04:55:00.128119 16868 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:00.128180 16868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:00.128341 16868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:00.130419 16868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "d719e8fcada44315b2fe4c2fc656fe7f"
format_stamp: "Formatted at 2026-08-08 04:55:00 on dist-test-slave-zpvx"
server_key: "a735142df7d108614fd15827e018d145"
server_key_iv: "73f77d4387a556ee11bf5f0b89586b23"
server_key_version: "encryptionkey@0"
I20260808 04:55:00.130959 16868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "d719e8fcada44315b2fe4c2fc656fe7f"
format_stamp: "Formatted at 2026-08-08 04:55:00 on dist-test-slave-zpvx"
server_key: "a735142df7d108614fd15827e018d145"
server_key_iv: "73f77d4387a556ee11bf5f0b89586b23"
server_key_version: "encryptionkey@0"
I20260808 04:55:00.134752 16868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:55:00.137329 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:00.138610 16868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:55:00.138756 16868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "d719e8fcada44315b2fe4c2fc656fe7f"
format_stamp: "Formatted at 2026-08-08 04:55:00 on dist-test-slave-zpvx"
server_key: "a735142df7d108614fd15827e018d145"
server_key_iv: "73f77d4387a556ee11bf5f0b89586b23"
server_key_version: "encryptionkey@0"
I20260808 04:55:00.138862 16868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:00.155516 16868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:00.158562 16868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:00.158774 16868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:00.166200 16868 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45451
I20260808 04:55:00.166226 16987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45451 every 8 connection(s)
I20260808 04:55:00.167157 16868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:00.169943 16989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:00.175220 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f: Bootstrap starting.
I20260808 04:55:00.177044 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 16868
I20260808 04:55:00.177196 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:00.177512 15481 external_mini_cluster.cc:1428] Setting key 8d1f3e07ddfb224b65fb720dca32fb6f
I20260808 04:55:00.177810 16989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:00.178607 16989 log.cc:824] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:00.180526 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f: No bootstrap required, opened a new log
I20260808 04:55:00.184577 16989 raft_consensus.cc:348] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45451 } }
I20260808 04:55:00.184846 16989 raft_consensus.cc:374] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:00.184903 16989 raft_consensus.cc:729] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d719e8fcada44315b2fe4c2fc656fe7f, State: Initialized, Role: FOLLOWER
I20260808 04:55:00.185740 16989 consensus_queue.cc:260] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45451 } }
I20260808 04:55:00.185930 16989 raft_consensus.cc:388] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:00.186013 16989 raft_consensus.cc:482] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:00.186126 16989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 0 FOLLOWER]: Advancing to term 1
Aug 08 04:55:00 dist-test-slave-zpvx krb5kdc[16848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:55:00.187042 16989 raft_consensus.cc:504] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45451 } }
I20260808 04:55:00.187374 16989 leader_election.cc:302] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d719e8fcada44315b2fe4c2fc656fe7f; no voters:
I20260808 04:55:00.187717 16989 leader_election.cc:288] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:00.187788 17000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:00.188014 17000 raft_consensus.cc:686] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [term 1 LEADER]: Becoming Leader. State: Replica: d719e8fcada44315b2fe4c2fc656fe7f, State: Running, Role: LEADER
I20260808 04:55:00.188315 17000 consensus_queue.cc:237] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45451 } }
I20260808 04:55:00.188989 16989 sys_catalog.cc:564] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:00.189491 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d719e8fcada44315b2fe4c2fc656fe7f. Latest consensus state: current_term: 1 leader_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45451 } } }
I20260808 04:55:00.189642 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:00.190428 17001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719e8fcada44315b2fe4c2fc656fe7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45451 } } }
I20260808 04:55:00.190604 17001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:00.193619 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:00.178676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:54849 (local address 127.0.0.1:45451)
0808 04:55:00.179242 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:00.179263 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:00.179309 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:55:00.179994 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:00.180011 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:00.180339 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:00.180770 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:00.181655 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:00.182661 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:00.183963 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:00.184496 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:00.184619 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:00.187572 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:00.187578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:00.187594 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:00.187621 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:00.189745 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:00.190446 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:00.190454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:00.190461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:00.190625 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:00.191398 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:00.191404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:00.191408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:00.191887 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:00.192146 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:00.192541 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 04:55:00.192834 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":135,"threads_started":1}
W20260808 04:55:00.194093 17024 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:00.194231 17024 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:00.194360 17007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:00.195281 17007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:00.197255 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:00.201337 17007 catalog_manager.cc:1347] Generated new cluster ID: 9154a0b26faf45aab56a0a8407e1f4a9
I20260808 04:55:00.201400 17007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:00.209081 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:00.199854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:56335 (local address 127.0.0.1:45451)
0808 04:55:00.199974 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:00.199979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:00.199991 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:55:00.200034 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:00.200036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:00.200072 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:00.200167 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:00.200932 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:00.201761 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:00.203033 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:00.203433 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:00.203501 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:00.205322 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:00.205326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:00.205329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:00.205354 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:00.207032 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:00.207592 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:00.207596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:00.207598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:00.207676 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:00.208036 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:00.208039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:00.208041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:00.208239 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:00.208351 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:00.208815 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 04:55:00.208910 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:55:00.214056 17007 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:00.215188 17007 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:00.235256 17007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d719e8fcada44315b2fe4c2fc656fe7f: Generated new TSK 0
I20260808 04:55:00.236094 17007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:00.246199 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 16868
W20260808 04:55:00.257498 16846 connection.cc:535] client connection to 127.0.0.1:45451 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45451): Connection reset by peer (error 104)
2026-08-08T04:55:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1649 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:00 dist-test-slave-zpvx krb5kdc[17040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:00 dist-test-slave-zpvx krb5kdc[17040](info): set up 2 sockets
Aug 08 04:55:00 dist-test-slave-zpvx krb5kdc[17040](info): commencing operation
krb5kdc: starting...
W20260808 04:55:01.502108 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:01 dist-test-slave-zpvx krb5kdc[17040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:55:01.627691 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:01.738467 17058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:01.738771 17058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:01.738847 17058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:01.742969 17058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:01.743054 17058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:01.743093 17058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:01.743150 17058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:01.743191 17058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:01.748293 17058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:01.749359 17058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:01.750368 17058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:01.759033 17072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:01.759096 17069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:01.759217 17066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:01.759232 17058 server_base.cc:1034] running on GCE node
I20260808 04:55:01.759914 17058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:01.761075 17058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:01.762269 17058 hybrid_clock.cc:648] HybridClock initialized: now 1786164901762256 us; error 27 us; skew 500 ppm
Aug 08 04:55:01 dist-test-slave-zpvx krb5kdc[17040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:01.765067 17058 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:55:01.765879 17058 webserver.cc:460] Webserver started at http://127.0.0.1:36027/ using document root <none> and password file <none>
I20260808 04:55:01.766381 17058 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:01.766436 17058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:01.766594 17058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:01.768720 17058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "a63b00989ad24414b835d7b8d08413db"
format_stamp: "Formatted at 2026-08-08 04:55:01 on dist-test-slave-zpvx"
server_key: "797f8c6d66b0956f75d0038d33451f7b"
server_key_iv: "16e2dc200f75a69f02bc76d4c525c744"
server_key_version: "encryptionkey@0"
I20260808 04:55:01.769266 17058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "a63b00989ad24414b835d7b8d08413db"
format_stamp: "Formatted at 2026-08-08 04:55:01 on dist-test-slave-zpvx"
server_key: "797f8c6d66b0956f75d0038d33451f7b"
server_key_iv: "16e2dc200f75a69f02bc76d4c525c744"
server_key_version: "encryptionkey@0"
I20260808 04:55:01.772959 17058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:55:01.775282 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:01.776547 17058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:55:01.776698 17058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "a63b00989ad24414b835d7b8d08413db"
format_stamp: "Formatted at 2026-08-08 04:55:01 on dist-test-slave-zpvx"
server_key: "797f8c6d66b0956f75d0038d33451f7b"
server_key_iv: "16e2dc200f75a69f02bc76d4c525c744"
server_key_version: "encryptionkey@0"
I20260808 04:55:01.776798 17058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:01.787003 17058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:01.789887 17058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:01.790164 17058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:01.797808 17058 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41443
I20260808 04:55:01.797830 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41443 every 8 connection(s)
I20260808 04:55:01.798807 17058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:01.801555 17168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:01.803817 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17058
I20260808 04:55:01.803965 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:01.804328 15481 external_mini_cluster.cc:1428] Setting key 5355a6474c9abf455ffa29a7196f3551
I20260808 04:55:01.807843 17168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db: Bootstrap starting.
I20260808 04:55:01.810364 17168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:01.811257 17168 log.cc:824] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:01.813297 17168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db: No bootstrap required, opened a new log
I20260808 04:55:01.813558 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:01.805731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:45001 (local address 127.0.0.1:41443)
0808 04:55:01.806128 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:01.806144 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:01.806180 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:55:01.806886 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:01.806905 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:01.807237 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:01.807508 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:01.807833 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:01.808893 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:01.810099 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:01.810657 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:01.810785 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:01.811794 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:01.811799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:01.811812 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:01.812196 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:01.812251 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:01.812275 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:01.812453 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:01.812659 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 04:55:01.812800 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
I20260808 04:55:01.816754 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:01.817544 17168 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b00989ad24414b835d7b8d08413db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260808 04:55:01.817762 17168 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:01.817821 17168 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63b00989ad24414b835d7b8d08413db, State: Initialized, Role: FOLLOWER
I20260808 04:55:01.818274 17168 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b00989ad24414b835d7b8d08413db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260808 04:55:01.818397 17168 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:01.818480 17168 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:01.818603 17168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:01.819785 17168 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b00989ad24414b835d7b8d08413db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260808 04:55:01.820114 17168 leader_election.cc:302] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63b00989ad24414b835d7b8d08413db; no voters:
I20260808 04:55:01.820454 17168 leader_election.cc:288] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:01.820554 17184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:01.820791 17184 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [term 1 LEADER]: Becoming Leader. State: Replica: a63b00989ad24414b835d7b8d08413db, State: Running, Role: LEADER
I20260808 04:55:01.821148 17184 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b00989ad24414b835d7b8d08413db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } }
I20260808 04:55:01.821676 17168 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:01.822441 17184 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63b00989ad24414b835d7b8d08413db. Latest consensus state: current_term: 1 leader_uuid: "a63b00989ad24414b835d7b8d08413db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b00989ad24414b835d7b8d08413db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } } }
I20260808 04:55:01.822626 17184 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:01.822888 17185 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63b00989ad24414b835d7b8d08413db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b00989ad24414b835d7b8d08413db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41443 } } }
I20260808 04:55:01.823063 17185 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:01.823100 17200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:01.825011 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:01.818783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:49445 (local address 127.0.0.1:41443)
0808 04:55:01.818912 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:01.818917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:01.819107 (+ 190us) server_negotiation.cc:407] Connection header received
0808 04:55:01.819208 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:01.819211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:01.819249 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:01.819312 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:01.819641 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:01.820190 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:01.821381 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:01.821760 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:01.821895 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:01.824526 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:01.824529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:01.824531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:01.824543 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:01.824567 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:01.824577 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:01.824685 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:01.824825 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 04:55:01.824860 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:55:01.826718 17200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:01.832413 17200 catalog_manager.cc:1347] Generated new cluster ID: d20c16f0d35e479d8c848d4c20a1afb7
I20260808 04:55:01.832501 17200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:01.840512 17200 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:01.841594 17200 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:01.865527 17200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63b00989ad24414b835d7b8d08413db: Generated new TSK 0
I20260808 04:55:01.866305 17200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:01.871454 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17058
W20260808 04:55:01.882918 17037 connection.cc:535] client connection to 127.0.0.1:41443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41443): Connection reset by peer (error 104)
2026-08-08T04:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1625 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:01 dist-test-slave-zpvx krb5kdc[17216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:01 dist-test-slave-zpvx krb5kdc[17216](info): set up 2 sockets
Aug 08 04:55:01 dist-test-slave-zpvx krb5kdc[17216](info): commencing operation
krb5kdc: starting...
W20260808 04:55:03.134079 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:03 dist-test-slave-zpvx krb5kdc[17216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:55:03.264128 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:03.375460 17239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:03.375727 17239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:03.375768 17239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:03.379804 17239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:03.379868 17239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:03.379886 17239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:03.379904 17239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:03.379920 17239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:03.385182 17239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:03.386229 17239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:03.387274 17239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:03.395744 17249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:03.395793 17239 server_base.cc:1034] running on GCE node
W20260808 04:55:03.395742 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:03.395751 17253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:03.396687 17239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:03.397708 17239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:03.398878 17239 hybrid_clock.cc:648] HybridClock initialized: now 1786164903398858 us; error 33 us; skew 500 ppm
Aug 08 04:55:03 dist-test-slave-zpvx krb5kdc[17216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:03.401757 17239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:55:03.402608 17239 webserver.cc:460] Webserver started at http://127.0.0.1:43089/ using document root <none> and password file <none>
I20260808 04:55:03.403162 17239 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:03.403254 17239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:03.403483 17239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:03.405540 17239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "d9a66f605e704dce889eec1f2a72475d"
format_stamp: "Formatted at 2026-08-08 04:55:03 on dist-test-slave-zpvx"
server_key: "50c26c843ef5d24af3e1ba25cc7fbd52"
server_key_iv: "5718a327f58375276241cf6bdc96462d"
server_key_version: "encryptionkey@0"
I20260808 04:55:03.406122 17239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "d9a66f605e704dce889eec1f2a72475d"
format_stamp: "Formatted at 2026-08-08 04:55:03 on dist-test-slave-zpvx"
server_key: "50c26c843ef5d24af3e1ba25cc7fbd52"
server_key_iv: "5718a327f58375276241cf6bdc96462d"
server_key_version: "encryptionkey@0"
I20260808 04:55:03.409904 17239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:55:03.412421 17264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:03.413691 17239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:55:03.413836 17239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "d9a66f605e704dce889eec1f2a72475d"
format_stamp: "Formatted at 2026-08-08 04:55:03 on dist-test-slave-zpvx"
server_key: "50c26c843ef5d24af3e1ba25cc7fbd52"
server_key_iv: "5718a327f58375276241cf6bdc96462d"
server_key_version: "encryptionkey@0"
I20260808 04:55:03.413947 17239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:03.418622 17239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:03.421550 17239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:03.421741 17239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:03.429188 17239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39705
I20260808 04:55:03.429208 17352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39705 every 8 connection(s)
I20260808 04:55:03.430064 17239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:03.430737 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17239
I20260808 04:55:03.430892 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:03.431198 15481 external_mini_cluster.cc:1428] Setting key 7ae846ae14dff860d9cb900fe6559778
I20260808 04:55:03.433473 17353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:03.436877 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:03.432614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:35555 (local address 127.0.0.1:39705)
0808 04:55:03.433040 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:03.433058 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:03.433101 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:55:03.433948 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:03.433971 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:03.434316 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:03.434669 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:03.435453 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:03.435459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:03.435473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:03.435486 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:03.435547 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:03.435572 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:03.435714 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:03.435795 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:55:03.435946 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260808 04:55:03.439718 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:03.440428 17353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d: Bootstrap starting.
I20260808 04:55:03.443147 17353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:03.443571 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:03.441833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:46699 (local address 127.0.0.1:39705)
0808 04:55:03.441943 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:03.441946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:03.442170 (+ 224us) server_negotiation.cc:407] Connection header received
0808 04:55:03.442279 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:03.442281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:03.442308 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:03.442373 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:03.443068 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:03.443071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:03.443073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:03.443083 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:03.443107 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:03.443117 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:03.443193 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:03.443371 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 04:55:03.443424 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 04:55:03.444016 17353 log.cc:824] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:03.445861 17353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d: No bootstrap required, opened a new log
I20260808 04:55:03.449327 17353 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a66f605e704dce889eec1f2a72475d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 04:55:03.449535 17353 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:03.449605 17353 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a66f605e704dce889eec1f2a72475d, State: Initialized, Role: FOLLOWER
I20260808 04:55:03.450042 17353 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a66f605e704dce889eec1f2a72475d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 04:55:03.450163 17353 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:03.450248 17353 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:03.450369 17353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:03.451205 17353 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a66f605e704dce889eec1f2a72475d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 04:55:03.451548 17353 leader_election.cc:302] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9a66f605e704dce889eec1f2a72475d; no voters:
I20260808 04:55:03.451844 17353 leader_election.cc:288] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:03.451993 17370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:03.452273 17370 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [term 1 LEADER]: Becoming Leader. State: Replica: d9a66f605e704dce889eec1f2a72475d, State: Running, Role: LEADER
I20260808 04:55:03.452592 17370 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a66f605e704dce889eec1f2a72475d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 04:55:03.453075 17353 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:03.454356 17372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9a66f605e704dce889eec1f2a72475d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a66f605e704dce889eec1f2a72475d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260808 04:55:03.454371 17373 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a66f605e704dce889eec1f2a72475d. Latest consensus state: current_term: 1 leader_uuid: "d9a66f605e704dce889eec1f2a72475d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a66f605e704dce889eec1f2a72475d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260808 04:55:03.454553 17373 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:03.454890 17372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d [sys.catalog]: This master's current role is: LEADER
W20260808 04:55:03.458083 17387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:03.458171 17387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:03.458257 17380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:03.458956 17380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:03.464270 17380 catalog_manager.cc:1347] Generated new cluster ID: 92e2546a2b194496b6376e321629c85d
I20260808 04:55:03.464337 17380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:03.486784 17380 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:03.487140 17380 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:03.495253 17380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9a66f605e704dce889eec1f2a72475d: Generated new TSK 0
I20260808 04:55:03.495931 17380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:03.500319 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17239
2026-08-08T04:55:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1630 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:03 dist-test-slave-zpvx krb5kdc[17397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:03 dist-test-slave-zpvx krb5kdc[17397](info): set up 2 sockets
Aug 08 04:55:03 dist-test-slave-zpvx krb5kdc[17397](info): commencing operation
krb5kdc: starting...
W20260808 04:55:04.764631 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:04 dist-test-slave-zpvx krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:04.890796 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33701
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:05.001950 17423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:05.002245 17423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:05.002322 17423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:05.006250 17423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:05.006342 17423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:05.006382 17423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:05.006440 17423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:05.006481 17423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:05.011569 17423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:05.012637 17423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:05.013631 17423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:05.022053 17432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:05.022117 17431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:05.022244 17423 server_base.cc:1034] running on GCE node
W20260808 04:55:05.022094 17435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:05.022912 17423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:05.024001 17423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:05.025192 17423 hybrid_clock.cc:648] HybridClock initialized: now 1786164905025170 us; error 31 us; skew 500 ppm
Aug 08 04:55:05 dist-test-slave-zpvx krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164905, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:05.027896 17423 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:05.028817 17423 webserver.cc:460] Webserver started at http://127.15.30.126:33925/ using document root <none> and password file <none>
I20260808 04:55:05.029378 17423 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:05.029471 17423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:05.029704 17423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:05.031674 17423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "8d4dc94df13447eba1df774ff33e851c"
format_stamp: "Formatted at 2026-08-08 04:55:05 on dist-test-slave-zpvx"
server_key: "327b2dfee597e1767af4991069aa22ad"
server_key_iv: "927028c201ea5d10b1054d81cd28112e"
server_key_version: "encryptionkey@0"
I20260808 04:55:05.032250 17423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "8d4dc94df13447eba1df774ff33e851c"
format_stamp: "Formatted at 2026-08-08 04:55:05 on dist-test-slave-zpvx"
server_key: "327b2dfee597e1767af4991069aa22ad"
server_key_iv: "927028c201ea5d10b1054d81cd28112e"
server_key_version: "encryptionkey@0"
I20260808 04:55:05.035825 17423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:55:05.038287 17445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:05.039481 17423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:55:05.039620 17423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "8d4dc94df13447eba1df774ff33e851c"
format_stamp: "Formatted at 2026-08-08 04:55:05 on dist-test-slave-zpvx"
server_key: "327b2dfee597e1767af4991069aa22ad"
server_key_iv: "927028c201ea5d10b1054d81cd28112e"
server_key_version: "encryptionkey@0"
I20260808 04:55:05.039740 17423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:05.054487 17423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:05.057421 17423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:05.057617 17423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:05.064541 17423 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33701
I20260808 04:55:05.064576 17524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33701 every 8 connection(s)
I20260808 04:55:05.065449 17423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:05.067317 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17423
I20260808 04:55:05.067469 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:05.067790 15481 external_mini_cluster.cc:1428] Setting key 185107d4cfbdcb5c50deb33a43800887
I20260808 04:55:05.068559 17525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:05.074299 17525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c: Bootstrap starting.
Aug 08 04:55:05 dist-test-slave-zpvx krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:55:05.076913 17525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:05.077737 17525 log.cc:824] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:05.079762 17525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c: No bootstrap required, opened a new log
I20260808 04:55:05.082170 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:05.069260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53367 (local address 127.15.30.126:33701)
0808 04:55:05.069735 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:05.069750 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:05.069786 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:55:05.070415 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:05.070430 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:05.070754 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:05.071086 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:05.071918 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:05.072961 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:05.073838 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:05.074304 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:05.074477 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:05.077353 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:05.077359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:05.077371 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:05.077399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:05.079177 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:05.079715 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:05.079720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:05.079724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:05.079883 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:05.080250 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:05.080253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:05.080255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:05.080704 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:05.080933 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:05.081260 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:55:05.081481 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":158,"threads_started":1}
I20260808 04:55:05.083850 17525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4dc94df13447eba1df774ff33e851c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33701 } }
I20260808 04:55:05.084106 17525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:05.084210 17525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d4dc94df13447eba1df774ff33e851c, State: Initialized, Role: FOLLOWER
I20260808 04:55:05.084728 17525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4dc94df13447eba1df774ff33e851c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33701 } }
I20260808 04:55:05.084880 17525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:05.084923 17525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:05.085016 17525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:05.085577 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:05.086081 17525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4dc94df13447eba1df774ff33e851c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33701 } }
I20260808 04:55:05.086412 17525 leader_election.cc:302] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d4dc94df13447eba1df774ff33e851c; no voters:
I20260808 04:55:05.086768 17525 leader_election.cc:288] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:05.087141 17546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:05.087355 17546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [term 1 LEADER]: Becoming Leader. State: Replica: 8d4dc94df13447eba1df774ff33e851c, State: Running, Role: LEADER
I20260808 04:55:05.088102 17525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:05.088665 17546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4dc94df13447eba1df774ff33e851c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33701 } }
I20260808 04:55:05.090327 17537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d4dc94df13447eba1df774ff33e851c. Latest consensus state: current_term: 1 leader_uuid: "8d4dc94df13447eba1df774ff33e851c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4dc94df13447eba1df774ff33e851c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33701 } } }
I20260808 04:55:05.090755 17537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:05.090545 17548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d4dc94df13447eba1df774ff33e851c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4dc94df13447eba1df774ff33e851c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33701 } } }
I20260808 04:55:05.090997 17548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c [sys.catalog]: This master's current role is: LEADER
W20260808 04:55:05.093588 17567 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:05.093672 17567 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:05.093765 17568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:05.094663 17568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:05.096828 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:05.088030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38677 (local address 127.15.30.126:33701)
0808 04:55:05.088124 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:05.088128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:05.088141 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:55:05.088178 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:05.088180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:05.088239 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:05.088325 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:05.089298 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:05.090116 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:05.091312 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:05.091624 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:05.091732 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:05.093387 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:05.093390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:05.093393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:05.093424 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:05.094956 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:05.095452 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:05.095458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:05.095464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:05.095550 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:05.095894 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:05.095898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:05.095899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:05.096086 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:05.096245 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:05.096558 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:55:05.096654 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 04:55:05.100455 17568 catalog_manager.cc:1347] Generated new cluster ID: e25de27b6eec4bc8b960d9711ecac5ba
I20260808 04:55:05.100572 17568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:05.121658 17568 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:05.122998 17568 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:05.128966 17568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d4dc94df13447eba1df774ff33e851c: Generated new TSK 0
I20260808 04:55:05.129801 17568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:05.148101 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17423
W20260808 04:55:05.163828 17394 connection.cc:535] client connection to 127.15.30.126:33701 recv error: Network error: failed to read from TLS socket (remote: 127.15.30.126:33701): Connection reset by peer (error 104)
2026-08-08T04:55:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1650 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:05 dist-test-slave-zpvx krb5kdc[17581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:05 dist-test-slave-zpvx krb5kdc[17581](info): set up 2 sockets
Aug 08 04:55:05 dist-test-slave-zpvx krb5kdc[17581](info): commencing operation
krb5kdc: starting...
W20260808 04:55:06.424615 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:06 dist-test-slave-zpvx krb5kdc[17581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:06.551852 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34435
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:06.665248 17602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:06.665511 17602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:06.665552 17602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:06.670774 17602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:06.670837 17602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:06.670858 17602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:06.670876 17602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:06.670890 17602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:06.676107 17602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:34435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:34435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:06.677150 17602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:06.678174 17602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:06.687093 17610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:06.687099 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:06.687276 17602 server_base.cc:1034] running on GCE node
W20260808 04:55:06.687348 17608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:06.688313 17602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:06.689769 17602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:06.690979 17602 hybrid_clock.cc:648] HybridClock initialized: now 1786164906690951 us; error 47 us; skew 500 ppm
Aug 08 04:55:06 dist-test-slave-zpvx krb5kdc[17581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164906, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:06.694991 17602 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:06.696261 17602 webserver.cc:460] Webserver started at http://127.15.30.126:36009/ using document root <none> and password file <none>
I20260808 04:55:06.696863 17602 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:06.696926 17602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:06.697086 17602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:06.699138 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "8698586ab21a4c409b7a261f02f2f68c"
format_stamp: "Formatted at 2026-08-08 04:55:06 on dist-test-slave-zpvx"
server_key: "f5f853941586480876231432caa98fa0"
server_key_iv: "7c9caa383d1898b96bf97d7c1e04f678"
server_key_version: "encryptionkey@0"
I20260808 04:55:06.699674 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "8698586ab21a4c409b7a261f02f2f68c"
format_stamp: "Formatted at 2026-08-08 04:55:06 on dist-test-slave-zpvx"
server_key: "f5f853941586480876231432caa98fa0"
server_key_iv: "7c9caa383d1898b96bf97d7c1e04f678"
server_key_version: "encryptionkey@0"
I20260808 04:55:06.703356 17602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:55:06.705785 17626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:06.707046 17602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:55:06.707165 17602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "8698586ab21a4c409b7a261f02f2f68c"
format_stamp: "Formatted at 2026-08-08 04:55:06 on dist-test-slave-zpvx"
server_key: "f5f853941586480876231432caa98fa0"
server_key_iv: "7c9caa383d1898b96bf97d7c1e04f678"
server_key_version: "encryptionkey@0"
I20260808 04:55:06.707247 17602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:06.725374 17602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:06.728598 17602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:06.728756 17602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:06.736115 17602 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:34435
I20260808 04:55:06.736140 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:34435 every 8 connection(s)
I20260808 04:55:06.737140 17602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:06.738579 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17602
I20260808 04:55:06.738729 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:06.739049 15481 external_mini_cluster.cc:1428] Setting key dfd279be3fac62225c093e18e083a58a
I20260808 04:55:06.740449 17718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:06.746122 17718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c: Bootstrap starting.
I20260808 04:55:06.748432 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:06.740580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42125 (local address 127.15.30.126:34435)
0808 04:55:06.741019 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:06.741035 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:06.741071 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:55:06.741735 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:06.741751 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:06.742074 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:06.742420 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:06.742817 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:06.743810 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:06.744758 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:06.745238 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:06.745428 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:06.746395 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:06.746401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:06.746415 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:06.746890 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:06.746956 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:06.746986 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:06.747127 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:06.747373 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 04:55:06.747514 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":138,"threads_started":1}
I20260808 04:55:06.748878 17718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:06.749742 17718 log.cc:824] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:06.751550 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:06.751967 17718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c: No bootstrap required, opened a new log
I20260808 04:55:06.755826 17718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698586ab21a4c409b7a261f02f2f68c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34435 } }
I20260808 04:55:06.756032 17718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:06.756132 17718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8698586ab21a4c409b7a261f02f2f68c, State: Initialized, Role: FOLLOWER
I20260808 04:55:06.756629 17718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698586ab21a4c409b7a261f02f2f68c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34435 } }
I20260808 04:55:06.756759 17718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:06.756842 17718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:06.756955 17718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:06.757848 17718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698586ab21a4c409b7a261f02f2f68c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34435 } }
I20260808 04:55:06.758003 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:06.753740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54393 (local address 127.15.30.126:34435)
0808 04:55:06.753867 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:06.753871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:06.754115 (+ 244us) server_negotiation.cc:407] Connection header received
0808 04:55:06.754265 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:06.754268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:06.754313 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:06.754395 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:06.754664 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:06.755174 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:06.755999 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:06.756438 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:06.756557 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:06.757471 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:06.757474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:06.757477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:06.757489 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:06.757521 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:06.757531 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:06.757626 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:06.757776 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 04:55:06.757812 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:55:06.758281 17718 leader_election.cc:302] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8698586ab21a4c409b7a261f02f2f68c; no voters:
I20260808 04:55:06.758631 17718 leader_election.cc:288] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:06.758718 17735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:06.758949 17735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [term 1 LEADER]: Becoming Leader. State: Replica: 8698586ab21a4c409b7a261f02f2f68c, State: Running, Role: LEADER
I20260808 04:55:06.759236 17735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698586ab21a4c409b7a261f02f2f68c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34435 } }
I20260808 04:55:06.759749 17718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:06.760345 17735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8698586ab21a4c409b7a261f02f2f68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698586ab21a4c409b7a261f02f2f68c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34435 } } }
I20260808 04:55:06.760499 17735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:06.760730 17735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8698586ab21a4c409b7a261f02f2f68c. Latest consensus state: current_term: 1 leader_uuid: "8698586ab21a4c409b7a261f02f2f68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698586ab21a4c409b7a261f02f2f68c" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 34435 } } }
I20260808 04:55:06.760821 17735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:06.761113 17745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:06.764108 17745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:06.769680 17745 catalog_manager.cc:1347] Generated new cluster ID: 9a5988bd2e8042acb1f93ca479c4007b
I20260808 04:55:06.769743 17745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:06.783322 17745 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:06.784196 17745 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:06.789706 17745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8698586ab21a4c409b7a261f02f2f68c: Generated new TSK 0
I20260808 04:55:06.790329 17745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:06.799973 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17602
W20260808 04:55:06.810552 17578 connection.cc:535] client connection to 127.15.30.126:34435 recv error: Network error: failed to read from TLS socket (remote: 127.15.30.126:34435): Connection reset by peer (error 104)
2026-08-08T04:55:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1646 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:06 dist-test-slave-zpvx krb5kdc[17763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:06 dist-test-slave-zpvx krb5kdc[17763](info): set up 2 sockets
Aug 08 04:55:06 dist-test-slave-zpvx krb5kdc[17763](info): commencing operation
krb5kdc: starting...
W20260808 04:55:08.047526 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:08 dist-test-slave-zpvx krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:08.175406 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43513
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:43513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:08.288347 17787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:08.288621 17787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:08.288662 17787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:08.292536 17787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:08.292587 17787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:08.292606 17787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:08.292623 17787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:08.292637 17787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:08.298280 17787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:43513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:43513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:08.299237 17787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:08.300179 17787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:08.308810 17801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:08.308751 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:08.308949 17787 server_base.cc:1034] running on GCE node
W20260808 04:55:08.308750 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:08.309728 17787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:08.310812 17787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:08.312013 17787 hybrid_clock.cc:648] HybridClock initialized: now 1786164908311973 us; error 53 us; skew 500 ppm
Aug 08 04:55:08 dist-test-slave-zpvx krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164908, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:08.314823 17787 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:08.315671 17787 webserver.cc:460] Webserver started at http://127.15.30.126:34101/ using document root <none> and password file <none>
I20260808 04:55:08.316233 17787 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:08.316324 17787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:08.316593 17787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:08.318610 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "134a8311f05c48739ef808a220758f86"
format_stamp: "Formatted at 2026-08-08 04:55:08 on dist-test-slave-zpvx"
server_key: "fb7ebc32347e9d952adbb254199db916"
server_key_iv: "f4faf7ad4b58741ff2456ba57ace895d"
server_key_version: "encryptionkey@0"
I20260808 04:55:08.319191 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "134a8311f05c48739ef808a220758f86"
format_stamp: "Formatted at 2026-08-08 04:55:08 on dist-test-slave-zpvx"
server_key: "fb7ebc32347e9d952adbb254199db916"
server_key_iv: "f4faf7ad4b58741ff2456ba57ace895d"
server_key_version: "encryptionkey@0"
I20260808 04:55:08.322978 17787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:55:08.325709 17811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:08.327149 17787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:55:08.327319 17787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "134a8311f05c48739ef808a220758f86"
format_stamp: "Formatted at 2026-08-08 04:55:08 on dist-test-slave-zpvx"
server_key: "fb7ebc32347e9d952adbb254199db916"
server_key_iv: "f4faf7ad4b58741ff2456ba57ace895d"
server_key_version: "encryptionkey@0"
I20260808 04:55:08.327436 17787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:08.332160 17787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:08.335093 17787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:08.335299 17787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:08.342800 17787 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:43513
I20260808 04:55:08.342829 17884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:43513 every 8 connection(s)
I20260808 04:55:08.343696 17787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:08.346644 17885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:08.351815 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17787
I20260808 04:55:08.351984 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:08.352341 15481 external_mini_cluster.cc:1428] Setting key d15496181e54b7bf00f1987e33b7933c
I20260808 04:55:08.353092 17885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86: Bootstrap starting.
I20260808 04:55:08.355692 17885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:08.356691 17885 log.cc:824] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:08.357978 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:08.353899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60069 (local address 127.15.30.126:43513)
0808 04:55:08.354375 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:08.354391 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:08.354429 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:55:08.355123 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:08.355141 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:08.355435 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:08.355801 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:08.356716 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:08.356722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:08.356734 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:08.356746 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:08.356802 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:08.356823 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:08.356962 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:08.357084 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:55:08.357259 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
I20260808 04:55:08.358969 17885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86: No bootstrap required, opened a new log
I20260808 04:55:08.361189 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:08.363240 17885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134a8311f05c48739ef808a220758f86" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43513 } }
I20260808 04:55:08.363545 17885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:08.363632 17885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134a8311f05c48739ef808a220758f86, State: Initialized, Role: FOLLOWER
I20260808 04:55:08.364149 17885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134a8311f05c48739ef808a220758f86" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43513 } }
I20260808 04:55:08.364351 17885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:08.364457 17885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:08.364591 17885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:08.365360 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:08.363430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53023 (local address 127.15.30.126:43513)
0808 04:55:08.363570 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:08.363574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:08.363837 (+ 263us) server_negotiation.cc:407] Connection header received
0808 04:55:08.364006 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:08.364009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:08.364046 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:08.364143 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:08.364973 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:08.364977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:08.364980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:08.364993 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:08.365035 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:08.365049 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:08.365156 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:08.365193 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 04:55:08.365243 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:55:08.365857 17885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134a8311f05c48739ef808a220758f86" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43513 } }
I20260808 04:55:08.366261 17885 leader_election.cc:302] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 134a8311f05c48739ef808a220758f86; no voters:
I20260808 04:55:08.366657 17885 leader_election.cc:288] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:08.366788 17897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:08.367051 17897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [term 1 LEADER]: Becoming Leader. State: Replica: 134a8311f05c48739ef808a220758f86, State: Running, Role: LEADER
I20260808 04:55:08.367372 17897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134a8311f05c48739ef808a220758f86" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43513 } }
I20260808 04:55:08.367973 17885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:08.369362 17900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "134a8311f05c48739ef808a220758f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134a8311f05c48739ef808a220758f86" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43513 } } }
I20260808 04:55:08.369379 17901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 134a8311f05c48739ef808a220758f86. Latest consensus state: current_term: 1 leader_uuid: "134a8311f05c48739ef808a220758f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "134a8311f05c48739ef808a220758f86" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 43513 } } }
I20260808 04:55:08.369513 17900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:08.369518 17901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:08.370020 17910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:08.373462 17910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:08.379637 17910 catalog_manager.cc:1347] Generated new cluster ID: 30c8116be4d14a079ec3f9ad78419b90
I20260808 04:55:08.379725 17910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:08.391274 17910 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:08.391613 17910 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:08.409505 17910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 134a8311f05c48739ef808a220758f86: Generated new TSK 0
I20260808 04:55:08.410264 17910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:08.422175 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17787
2026-08-08T04:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:08 dist-test-slave-zpvx krb5kdc[17939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:08 dist-test-slave-zpvx krb5kdc[17939](info): set up 2 sockets
Aug 08 04:55:08 dist-test-slave-zpvx krb5kdc[17939](info): commencing operation
krb5kdc: starting...
W20260808 04:55:09.670615 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:09 dist-test-slave-zpvx krb5kdc[17939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:09.799881 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33269
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:09.908936 17956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:09.909231 17956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:09.909317 17956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:09.913219 17956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:09.913309 17956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:09.913348 17956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:09.913405 17956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:09.913445 17956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:09.918545 17956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:09.919536 17956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:09.920552 17956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:09.928617 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:09.928681 17956 server_base.cc:1034] running on GCE node
W20260808 04:55:09.928596 17972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:09.929010 17968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:09.929575 17956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:09.930617 17956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:09.931818 17956 hybrid_clock.cc:648] HybridClock initialized: now 1786164909931799 us; error 31 us; skew 500 ppm
Aug 08 04:55:09 dist-test-slave-zpvx krb5kdc[17939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164909, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:09.934597 17956 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:09.935467 17956 webserver.cc:460] Webserver started at http://127.15.30.126:45949/ using document root <none> and password file <none>
I20260808 04:55:09.936004 17956 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:09.936094 17956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:09.936339 17956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:09.938350 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "49b656362d31459b8ea2a37aee638689"
format_stamp: "Formatted at 2026-08-08 04:55:09 on dist-test-slave-zpvx"
server_key: "9d154e7f0b33416cf1ba01b320b187ac"
server_key_iv: "be6bbfb3a6ad22ed0f4e966e94c0f75d"
server_key_version: "encryptionkey@0"
I20260808 04:55:09.938913 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "49b656362d31459b8ea2a37aee638689"
format_stamp: "Formatted at 2026-08-08 04:55:09 on dist-test-slave-zpvx"
server_key: "9d154e7f0b33416cf1ba01b320b187ac"
server_key_iv: "be6bbfb3a6ad22ed0f4e966e94c0f75d"
server_key_version: "encryptionkey@0"
I20260808 04:55:09.942525 17956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 04:55:09.944898 17986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:09.946087 17956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:55:09.946230 17956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "49b656362d31459b8ea2a37aee638689"
format_stamp: "Formatted at 2026-08-08 04:55:09 on dist-test-slave-zpvx"
server_key: "9d154e7f0b33416cf1ba01b320b187ac"
server_key_iv: "be6bbfb3a6ad22ed0f4e966e94c0f75d"
server_key_version: "encryptionkey@0"
I20260808 04:55:09.946349 17956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:09.961181 17956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:09.964417 17956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:09.964613 17956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:09.971846 17956 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33269
I20260808 04:55:09.971856 18061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33269 every 8 connection(s)
I20260808 04:55:09.972817 17956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:09.975575 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:09.975631 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 17956
I20260808 04:55:09.975767 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:09.976090 15481 external_mini_cluster.cc:1428] Setting key b73f645521196b46db902b990a9bad86
I20260808 04:55:09.982609 18064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689: Bootstrap starting.
Aug 08 04:55:09 dist-test-slave-zpvx krb5kdc[17939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:55:09.985447 18064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:09.986315 18064 log.cc:824] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:09.988641 18064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689: No bootstrap required, opened a new log
I20260808 04:55:09.991065 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:09.977453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54237 (local address 127.15.30.126:33269)
0808 04:55:09.977870 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:09.977896 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:09.977932 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:55:09.978687 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:09.978705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:09.979049 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:09.979322 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:09.980234 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:09.981275 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:09.982437 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:09.982948 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:09.983064 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:09.985899 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:09.985905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:09.985920 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:09.985952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:09.987962 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:09.988424 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:09.988430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:09.988435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:09.988571 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:09.988899 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:09.988902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:09.988904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:09.989407 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:09.989611 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:09.989807 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 04:55:09.990086 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260808 04:55:09.993081 18064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b656362d31459b8ea2a37aee638689" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33269 } }
I20260808 04:55:09.993304 18064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:09.993382 18064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b656362d31459b8ea2a37aee638689, State: Initialized, Role: FOLLOWER
I20260808 04:55:09.993782 18064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b656362d31459b8ea2a37aee638689" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33269 } }
I20260808 04:55:09.993992 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:09.993996 18064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:09.994086 18064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:09.994216 18064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:09.995076 18064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b656362d31459b8ea2a37aee638689" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33269 } }
I20260808 04:55:09.995469 18064 leader_election.cc:302] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49b656362d31459b8ea2a37aee638689; no voters:
I20260808 04:55:09.995826 18064 leader_election.cc:288] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:09.995944 18085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:09.996310 18085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [term 1 LEADER]: Becoming Leader. State: Replica: 49b656362d31459b8ea2a37aee638689, State: Running, Role: LEADER
I20260808 04:55:09.996665 18085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b656362d31459b8ea2a37aee638689" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33269 } }
I20260808 04:55:09.997412 18064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:09.997848 18085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49b656362d31459b8ea2a37aee638689. Latest consensus state: current_term: 1 leader_uuid: "49b656362d31459b8ea2a37aee638689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b656362d31459b8ea2a37aee638689" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33269 } } }
I20260808 04:55:09.998594 18085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:09.998479 18087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49b656362d31459b8ea2a37aee638689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b656362d31459b8ea2a37aee638689" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33269 } } }
I20260808 04:55:09.998916 18087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:10.000130 18101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:10.003427 18101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:10.004899 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:09.996026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44347 (local address 127.15.30.126:33269)
0808 04:55:09.996128 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:09.996132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:09.996344 (+ 212us) server_negotiation.cc:407] Connection header received
0808 04:55:09.996544 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:09.996547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:09.996591 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:09.996678 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:09.997457 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:09.998204 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:09.999401 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:10.000294 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:10.000428 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:10.001692 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:10.001696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:10.001698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:10.001722 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:10.003172 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:10.003641 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:10.003645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:10.003647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:10.003729 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:10.004013 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:10.004016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:10.004018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:10.004187 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:10.004346 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:10.004624 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 04:55:10.004740 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 04:55:10.008826 18101 catalog_manager.cc:1347] Generated new cluster ID: 87db9c641fac41e887f11fefcf37ae52
I20260808 04:55:10.008890 18101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:10.019327 18101 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:10.020119 18101 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:10.029560 18101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49b656362d31459b8ea2a37aee638689: Generated new TSK 0
I20260808 04:55:10.030170 18101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:10.037364 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 17956
2026-08-08T04:55:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1610 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:10 dist-test-slave-zpvx krb5kdc[18119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:10 dist-test-slave-zpvx krb5kdc[18119](info): set up 2 sockets
Aug 08 04:55:10 dist-test-slave-zpvx krb5kdc[18119](info): commencing operation
krb5kdc: starting...
W20260808 04:55:11.371011 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:11 dist-test-slave-zpvx krb5kdc[18119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:11.501276 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33425
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:11.613461 18134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:11.613761 18134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:11.613837 18134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:11.617866 18134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:11.617949 18134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:11.617986 18134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:11.618042 18134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:11.618077 18134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:11.623179 18134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:11.624229 18134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:11.625329 18134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:11.634219 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:11.634234 18143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:11.634234 18145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:11.634343 18134 server_base.cc:1034] running on GCE node
I20260808 04:55:11.635082 18134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:11.636154 18134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:11.637357 18134 hybrid_clock.cc:648] HybridClock initialized: now 1786164911637301 us; error 88 us; skew 500 ppm
Aug 08 04:55:11 dist-test-slave-zpvx krb5kdc[18119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164911, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:11.646430 18134 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:11.647372 18134 webserver.cc:460] Webserver started at http://127.15.30.126:40523/ using document root <none> and password file <none>
I20260808 04:55:11.647964 18134 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:11.648061 18134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:11.648305 18134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:11.650432 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "2cd2c929754646c294aedfde209f8b12"
format_stamp: "Formatted at 2026-08-08 04:55:11 on dist-test-slave-zpvx"
server_key: "7aeccb3062d95cd8259445763d9fe1c5"
server_key_iv: "36ec94e377e4f0432ffbc1e20b82a4f2"
server_key_version: "encryptionkey@0"
I20260808 04:55:11.651051 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "2cd2c929754646c294aedfde209f8b12"
format_stamp: "Formatted at 2026-08-08 04:55:11 on dist-test-slave-zpvx"
server_key: "7aeccb3062d95cd8259445763d9fe1c5"
server_key_iv: "36ec94e377e4f0432ffbc1e20b82a4f2"
server_key_version: "encryptionkey@0"
I20260808 04:55:11.654884 18134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:55:11.657450 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:11.658725 18134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:55:11.658895 18134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "2cd2c929754646c294aedfde209f8b12"
format_stamp: "Formatted at 2026-08-08 04:55:11 on dist-test-slave-zpvx"
server_key: "7aeccb3062d95cd8259445763d9fe1c5"
server_key_iv: "36ec94e377e4f0432ffbc1e20b82a4f2"
server_key_version: "encryptionkey@0"
I20260808 04:55:11.659011 18134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:11.679222 18134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:11.682214 18134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:11.682420 18134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:11.690418 18134 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33425
I20260808 04:55:11.690475 18245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33425 every 8 connection(s)
I20260808 04:55:11.691622 18134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:11.694406 18246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:11.698151 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18134
I20260808 04:55:11.698325 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:11.698621 15481 external_mini_cluster.cc:1428] Setting key 50c6e11a48f376f20fbe6f5c17b5cbef
I20260808 04:55:11.700309 18246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12: Bootstrap starting.
I20260808 04:55:11.702661 18246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:11.703465 18246 log.cc:824] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:11.705696 18246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12: No bootstrap required, opened a new log
I20260808 04:55:11.707801 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:11.700064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37477 (local address 127.15.30.126:33425)
0808 04:55:11.700495 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:11.700511 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:11.700547 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:55:11.701173 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:11.701189 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:11.701507 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:11.701783 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:11.702183 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:11.703224 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:11.704364 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:11.704890 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:11.705026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:11.705886 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:11.705892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:11.705905 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:11.706314 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:11.706366 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:11.706389 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:11.706465 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:11.706638 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 04:55:11.706868 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260808 04:55:11.708811 18246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd2c929754646c294aedfde209f8b12" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33425 } }
I20260808 04:55:11.709110 18246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:11.709170 18246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2c929754646c294aedfde209f8b12, State: Initialized, Role: FOLLOWER
I20260808 04:55:11.709621 18246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd2c929754646c294aedfde209f8b12" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33425 } }
I20260808 04:55:11.709805 18246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:11.709878 18246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:11.710003 18246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:11.710914 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:11.711051 18246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd2c929754646c294aedfde209f8b12" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33425 } }
I20260808 04:55:11.711382 18246 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd2c929754646c294aedfde209f8b12; no voters:
I20260808 04:55:11.711678 18246 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:11.711819 18251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:11.712123 18251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd2c929754646c294aedfde209f8b12, State: Running, Role: LEADER
I20260808 04:55:11.712563 18251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd2c929754646c294aedfde209f8b12" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33425 } }
I20260808 04:55:11.712939 18246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:11.714035 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cd2c929754646c294aedfde209f8b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd2c929754646c294aedfde209f8b12" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33425 } } }
I20260808 04:55:11.714206 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:11.714991 18256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd2c929754646c294aedfde209f8b12. Latest consensus state: current_term: 1 leader_uuid: "2cd2c929754646c294aedfde209f8b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd2c929754646c294aedfde209f8b12" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33425 } } }
I20260808 04:55:11.715106 18256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12 [sys.catalog]: This master's current role is: LEADER
W20260808 04:55:11.718086 18283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:11.718204 18283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:11.718309 18277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:11.718703 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:11.713268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43601 (local address 127.15.30.126:33425)
0808 04:55:11.713377 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:11.713382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:11.713942 (+ 560us) server_negotiation.cc:407] Connection header received
0808 04:55:11.714171 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:11.714175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:11.714230 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:11.714322 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:11.714745 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:11.715605 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:11.716718 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:11.717028 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:11.717127 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:11.718082 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:11.718086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:11.718089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:11.718103 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:11.718131 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:11.718144 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:11.718217 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:11.718537 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 04:55:11.718600 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 04:55:11.719287 18277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:11.725099 18277 catalog_manager.cc:1347] Generated new cluster ID: 7344ab0c72f04f5592a60be89e13727d
I20260808 04:55:11.725178 18277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:11.748427 18277 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:11.749380 18277 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:11.756958 18277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cd2c929754646c294aedfde209f8b12: Generated new TSK 0
I20260808 04:55:11.757704 18277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:11.761868 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18134
2026-08-08T04:55:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1724 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:11 dist-test-slave-zpvx krb5kdc[18290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:11 dist-test-slave-zpvx krb5kdc[18290](info): set up 2 sockets
Aug 08 04:55:11 dist-test-slave-zpvx krb5kdc[18290](info): commencing operation
krb5kdc: starting...
W20260808 04:55:13.023906 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:13 dist-test-slave-zpvx krb5kdc[18290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:13.146837 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:39007
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:39007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:13.254860 18309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:13.255093 18309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:13.255133 18309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:13.258929 18309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:13.258973 18309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:13.258992 18309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:13.259007 18309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:13.259022 18309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:13.263921 18309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:39007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:39007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:13.264842 18309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:13.265764 18309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:13.274063 18316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:13.274072 18321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:13.274096 18317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:13.274256 18309 server_base.cc:1034] running on GCE node
I20260808 04:55:13.274804 18309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:13.275878 18309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:13.277041 18309 hybrid_clock.cc:648] HybridClock initialized: now 1786164913277029 us; error 37 us; skew 500 ppm
Aug 08 04:55:13 dist-test-slave-zpvx krb5kdc[18290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164913, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:13.279692 18309 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:13.280548 18309 webserver.cc:460] Webserver started at http://127.15.30.126:34435/ using document root <none> and password file <none>
I20260808 04:55:13.281054 18309 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:13.281119 18309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:13.281275 18309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:13.283212 18309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "e218e1d898284f5287b3e1c1ba6ad472"
format_stamp: "Formatted at 2026-08-08 04:55:13 on dist-test-slave-zpvx"
server_key: "61678a48a93f452a69986c85adec6132"
server_key_iv: "30594ae73d67d181cd14121e44fb5733"
server_key_version: "encryptionkey@0"
I20260808 04:55:13.283708 18309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "e218e1d898284f5287b3e1c1ba6ad472"
format_stamp: "Formatted at 2026-08-08 04:55:13 on dist-test-slave-zpvx"
server_key: "61678a48a93f452a69986c85adec6132"
server_key_iv: "30594ae73d67d181cd14121e44fb5733"
server_key_version: "encryptionkey@0"
I20260808 04:55:13.287293 18309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:55:13.289902 18332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:13.291102 18309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:55:13.291213 18309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "e218e1d898284f5287b3e1c1ba6ad472"
format_stamp: "Formatted at 2026-08-08 04:55:13 on dist-test-slave-zpvx"
server_key: "61678a48a93f452a69986c85adec6132"
server_key_iv: "30594ae73d67d181cd14121e44fb5733"
server_key_version: "encryptionkey@0"
I20260808 04:55:13.291294 18309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:13.295675 18309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:13.298791 18309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:13.298933 18309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:13.306340 18309 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:39007
I20260808 04:55:13.306362 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:39007 every 8 connection(s)
I20260808 04:55:13.307190 18309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:13.309854 18414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:13.312722 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18309
I20260808 04:55:13.312884 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:13.313180 15481 external_mini_cluster.cc:1428] Setting key 4b4da06283156f0043b246af87c64b18
I20260808 04:55:13.315569 18414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472: Bootstrap starting.
I20260808 04:55:13.318260 18414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:13.318487 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:13.314520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58419 (local address 127.15.30.126:39007)
0808 04:55:13.314932 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:13.314950 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:13.314989 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:55:13.315788 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:13.315813 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:13.316165 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:13.316550 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:13.317207 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:13.317212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:13.317226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:13.317237 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:13.317289 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:13.317311 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:13.317436 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:13.317497 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 04:55:13.317632 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":97,"threads_started":1}
I20260808 04:55:13.319149 18414 log.cc:824] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:13.321210 18414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472: No bootstrap required, opened a new log
I20260808 04:55:13.321260 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:13.324888 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:13.323275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40291 (local address 127.15.30.126:39007)
0808 04:55:13.323388 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:13.323392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:13.323523 (+ 131us) server_negotiation.cc:407] Connection header received
0808 04:55:13.323672 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:13.323675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:13.323707 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:13.323801 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:13.324505 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:13.324508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:13.324511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:55:13.324522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:13.324548 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:55:13.324567 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:13.324667 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:13.324743 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 04:55:13.324781 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:55:13.325033 18414 raft_consensus.cc:348] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e218e1d898284f5287b3e1c1ba6ad472" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 39007 } }
I20260808 04:55:13.325304 18414 raft_consensus.cc:374] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:55:13.325376 18414 raft_consensus.cc:729] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218e1d898284f5287b3e1c1ba6ad472, State: Initialized, Role: FOLLOWER
I20260808 04:55:13.325878 18414 consensus_queue.cc:260] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e218e1d898284f5287b3e1c1ba6ad472" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 39007 } }
I20260808 04:55:13.326023 18414 raft_consensus.cc:388] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:55:13.326112 18414 raft_consensus.cc:482] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:55:13.326205 18414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:13.327014 18414 raft_consensus.cc:504] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e218e1d898284f5287b3e1c1ba6ad472" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 39007 } }
I20260808 04:55:13.327340 18414 leader_election.cc:302] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e218e1d898284f5287b3e1c1ba6ad472; no voters:
I20260808 04:55:13.327636 18414 leader_election.cc:288] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:55:13.327714 18428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:13.327986 18428 raft_consensus.cc:686] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [term 1 LEADER]: Becoming Leader. State: Replica: e218e1d898284f5287b3e1c1ba6ad472, State: Running, Role: LEADER
I20260808 04:55:13.328414 18428 consensus_queue.cc:237] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e218e1d898284f5287b3e1c1ba6ad472" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 39007 } }
I20260808 04:55:13.328941 18414 sys_catalog.cc:564] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:13.330051 18429 sys_catalog.cc:455] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e218e1d898284f5287b3e1c1ba6ad472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e218e1d898284f5287b3e1c1ba6ad472" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 39007 } } }
I20260808 04:55:13.330153 18429 sys_catalog.cc:458] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:13.330583 18441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:13.330571 18431 sys_catalog.cc:455] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e218e1d898284f5287b3e1c1ba6ad472. Latest consensus state: current_term: 1 leader_uuid: "e218e1d898284f5287b3e1c1ba6ad472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e218e1d898284f5287b3e1c1ba6ad472" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 39007 } } }
I20260808 04:55:13.330765 18431 sys_catalog.cc:458] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:13.333724 18441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:13.339363 18441 catalog_manager.cc:1347] Generated new cluster ID: 744192c01a3e4214812b3d7194de68cc
I20260808 04:55:13.339419 18441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:13.347574 18441 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:13.347852 18441 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:13.355155 18441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e218e1d898284f5287b3e1c1ba6ad472: Generated new TSK 0
I20260808 04:55:13.355822 18441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:55:13.365247 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18309
2026-08-08T04:55:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1599 ms)
[----------] 18 tests from AuthTokenIssuingTest (29371 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:13 dist-test-slave-zpvx krb5kdc[18460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:13 dist-test-slave-zpvx krb5kdc[18460](info): set up 2 sockets
Aug 08 04:55:13 dist-test-slave-zpvx krb5kdc[18460](info): commencing operation
krb5kdc: starting...
W20260808 04:55:14.635798 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:14 dist-test-slave-zpvx krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:14.764037 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33395
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33395,127.15.30.125:44629,127.15.30.124:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:14.873936 18479 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:55:14.874275 18479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:14.874354 18479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:14.874433 18479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:14.878388 18479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:14.878477 18479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:14.878513 18479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:14.878556 18479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:14.878587 18479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:14.883674 18479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33395,127.15.30.125:44629,127.15.30.124:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:14.884758 18479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:14.885762 18479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:14.894163 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:14.894233 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:14.894318 18479 server_base.cc:1034] running on GCE node
W20260808 04:55:14.894249 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:14.895042 18479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:14.896075 18479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:14.897271 18479 hybrid_clock.cc:648] HybridClock initialized: now 1786164914897262 us; error 31 us; skew 500 ppm
Aug 08 04:55:14 dist-test-slave-zpvx krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:14.900161 18479 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:14.901002 18479 webserver.cc:460] Webserver started at http://127.15.30.126:39085/ using document root <none> and password file <none>
I20260808 04:55:14.901492 18479 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:14.901551 18479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:14.901712 18479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:14.903615 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "1184e0cf1063406095ea18e52df8ee72"
format_stamp: "Formatted at 2026-08-08 04:55:14 on dist-test-slave-zpvx"
server_key: "386fe6061df6e90ec2a67f89576677f2"
server_key_iv: "db477d13a62b4ef71193535a6c1b6ed8"
server_key_version: "encryptionkey@0"
I20260808 04:55:14.904127 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "1184e0cf1063406095ea18e52df8ee72"
format_stamp: "Formatted at 2026-08-08 04:55:14 on dist-test-slave-zpvx"
server_key: "386fe6061df6e90ec2a67f89576677f2"
server_key_iv: "db477d13a62b4ef71193535a6c1b6ed8"
server_key_version: "encryptionkey@0"
I20260808 04:55:14.907719 18479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 04:55:14.910120 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:14.911278 18479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:55:14.911375 18479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "1184e0cf1063406095ea18e52df8ee72"
format_stamp: "Formatted at 2026-08-08 04:55:14 on dist-test-slave-zpvx"
server_key: "386fe6061df6e90ec2a67f89576677f2"
server_key_iv: "db477d13a62b4ef71193535a6c1b6ed8"
server_key_version: "encryptionkey@0"
I20260808 04:55:14.911456 18479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:14.929001 18479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:14.932073 18479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:14.932219 18479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:14.939244 18479 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:33395
I20260808 04:55:14.939548 18580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:33395 every 8 connection(s)
I20260808 04:55:14.940198 18479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:14.943044 18581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:14.946988 18581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:14.950037 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18479
I20260808 04:55:14.950209 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:14.950490 15481 external_mini_cluster.cc:1428] Setting key 1245cc2c37dcc324e88c55a37d4c5dd8
Aug 08 04:55:14 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.30.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.125@KRBTEST.COM" created.
I20260808 04:55:14.963054 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:14.949137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.15.30.126:33395)
0808 04:55:14.949576 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:14.949591 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:14.949750 (+ 159us) server_negotiation.cc:407] Connection header received
0808 04:55:14.950925 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:14.950942 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:14.951183 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:14.951394 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:14.952302 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:14.953364 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:14.954890 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:14.955299 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:14.955420 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:14.958225 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:14.958231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:14.958240 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:14.958261 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:14.959815 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:14.960309 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:14.960321 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:14.960322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:14.960471 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:14.960812 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:14.960816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:14.960817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:14.961326 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:14.961471 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:14.961828 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 04:55:14.962149 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":310,"spinlock_wait_cycles":20352,"thread_start_us":162,"threads_started":1}
I20260808 04:55:14.963054 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:14.949136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33395 (local address 127.0.0.1:59218)
0808 04:55:14.949583 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:14.949639 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:14.950561 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:14.951432 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:14.951437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:14.951769 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:14.952096 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:14.952109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:14.953503 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:14.953508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:14.954742 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:14.954758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:14.955482 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:14.955491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:14.955671 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:14.956256 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:14.956279 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:14.958007 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:14.959934 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:14.959941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:14.959950 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:14.960199 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:14.960590 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:14.960593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:14.960594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:14.960712 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:14.961443 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:14.961449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:14.961588 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 04:55:14.961752 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 04:55:14.962044 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":57,"thread_start_us":130,"threads_started":1}
I20260808 04:55:14.965518 18581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:14.966187 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:14.965939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52886)
0808 04:55:14.966051 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:14.966133 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:55:14.966394 18496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
I20260808 04:55:14.967206 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:14.966990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52888)
0808 04:55:14.967095 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:14.967150 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:55:14.967372 18581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:44629: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab.
Entry for principal kudu/127.15.30.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab.
I20260808 04:55:14.979148 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:44629
--webserver_interface=127.15.30.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab
--principal=kudu/127.15.30.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33395,127.15.30.125:44629,127.15.30.124:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:55:15.016578 18581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } attempt: 1
I20260808 04:55:15.017503 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.017193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52904)
0808 04:55:15.017355 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.017421 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 04:55:15.018384 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.018170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52910)
0808 04:55:15.018277 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.018324 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:55:15.018559 18581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:44629: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
I20260808 04:55:15.086728 18581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } attempt: 2
I20260808 04:55:15.087649 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.087330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52916)
0808 04:55:15.087492 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.087559 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 04:55:15.088508 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.088178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52920)
0808 04:55:15.088308 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.088356 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 04:55:15.088688 18581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:44629: Network error: Client connection negotiation failed: client connection to 127.15.30.125:44629: connect: Connection refused (error 111)
W20260808 04:55:15.091717 18594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:55:15.091953 18594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:15.091975 18594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:15.092011 18594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:15.095973 18594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:15.096021 18594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:15.096042 18594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:15.096058 18594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:15.096073 18594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:15.101177 18594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33395,127.15.30.125:44629,127.15.30.124:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab
--principal=kudu/127.15.30.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.30.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:15.102082 18594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:15.102981 18594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:15.111234 18602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:15.111382 18594 server_base.cc:1034] running on GCE node
W20260808 04:55:15.111241 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:15.111253 18600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:15.112118 18594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:15.113205 18594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:15.114401 18594 hybrid_clock.cc:648] HybridClock initialized: now 1786164915114369 us; error 44 us; skew 500 ppm
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:15.117434 18594 init.cc:377] Logged in from keytab as kudu/127.15.30.125@KRBTEST.COM (short username kudu)
I20260808 04:55:15.118321 18594 webserver.cc:460] Webserver started at http://127.15.30.125:40309/ using document root <none> and password file <none>
I20260808 04:55:15.118870 18594 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:15.118961 18594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:15.119199 18594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:15.121290 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data/instance:
uuid: "59cd428d94f94b4c8a9955413b5a563a"
format_stamp: "Formatted at 2026-08-08 04:55:15 on dist-test-slave-zpvx"
server_key: "18ff428f54708b9a5761bfdc16bc56e8"
server_key_iv: "103250d5e21e2bccd10335ab0ce8a351"
server_key_version: "encryptionkey@0"
I20260808 04:55:15.121857 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/wal/instance:
uuid: "59cd428d94f94b4c8a9955413b5a563a"
format_stamp: "Formatted at 2026-08-08 04:55:15 on dist-test-slave-zpvx"
server_key: "18ff428f54708b9a5761bfdc16bc56e8"
server_key_iv: "103250d5e21e2bccd10335ab0ce8a351"
server_key_version: "encryptionkey@0"
I20260808 04:55:15.127040 18594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 04:55:15.129493 18616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:15.130693 18594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:55:15.130846 18594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/wal
uuid: "59cd428d94f94b4c8a9955413b5a563a"
format_stamp: "Formatted at 2026-08-08 04:55:15 on dist-test-slave-zpvx"
server_key: "18ff428f54708b9a5761bfdc16bc56e8"
server_key_iv: "103250d5e21e2bccd10335ab0ce8a351"
server_key_version: "encryptionkey@0"
I20260808 04:55:15.130959 18594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:15.141300 18594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:15.144881 18594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:15.145072 18594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:15.152143 18594 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.125:44629
I20260808 04:55:15.152536 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.125:44629 every 8 connection(s)
I20260808 04:55:15.153139 18594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
I20260808 04:55:15.155807 18693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:15.156643 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18594
I20260808 04:55:15.156798 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-1/wal/instance
I20260808 04:55:15.157090 15481 external_mini_cluster.cc:1428] Setting key 32d568a57e5aa1b07d4b95f63c967cc2
I20260808 04:55:15.159576 18693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.30.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.124@KRBTEST.COM" created.
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:55:15.174049 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.161527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.15.30.126:33395)
0808 04:55:15.161719 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.161727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.162483 (+ 756us) server_negotiation.cc:407] Connection header received
0808 04:55:15.163404 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.163411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.163479 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.163597 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.164784 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.165657 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.166837 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.167121 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.167214 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.169839 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.169847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.169849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.169873 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.171940 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.172487 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.172491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.172493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.172575 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.172927 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.172930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.172932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.173116 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.173226 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.173750 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.173871 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:55:15.174798 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.161852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33395 (local address 127.0.0.1:59232)
0808 04:55:15.162314 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.162366 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.163193 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.163742 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.163748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.164063 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.164634 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.164647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.165797 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.165799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.166716 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.166726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.167277 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.167280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.167424 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.167936 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.167951 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.169668 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.172064 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.172070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.172084 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.172339 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.172672 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.172677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.172679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.172823 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.173224 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.173231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.173474 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.173669 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.173975 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":131,"threads_started":1}
I20260808 04:55:15.176198 18693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:15.179869 18581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } attempt: 3
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab.
Entry for principal kudu/127.15.30.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab.
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
I20260808 04:55:15.185783 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:39437
--webserver_interface=127.15.30.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab
--principal=kudu/127.15.30.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:33395,127.15.30.125:44629,127.15.30.124:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:55:15.187657 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.176698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52934)
0808 04:55:15.176811 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.176826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.176895 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.177420 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.177422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.177650 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.177746 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.177752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.178774 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.178778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.179538 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.179548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.180300 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.180308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.180441 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.180915 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.180930 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.182189 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.184425 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.184429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.184431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.184631 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.184931 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.184935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.184937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.185019 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.186648 (+ 1629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.186654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.186767 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.186916 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.187089 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 04:55:15.187763 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.176823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.15.30.125:44629)
0808 04:55:15.177107 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.177117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.177135 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:55:15.177165 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.177170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.177304 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.177441 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.177853 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.178656 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.179660 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.180159 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.180285 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.182346 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.182359 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.182373 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.182423 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.184283 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.184749 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.184755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.184757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.184835 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.185138 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.185143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.185144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.185433 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.185575 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.187049 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.187202 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":105,"threads_started":1}
I20260808 04:55:15.189985 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.180284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:52950)
0808 04:55:15.180477 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.180491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.180574 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.180906 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.180912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.181136 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.181252 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.181260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.182210 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.182214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.182881 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.182891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.183373 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.183376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.183481 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.183877 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.183888 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.185448 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.188404 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.188408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.188410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.188647 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.188970 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.188974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.188979 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.189100 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.189486 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.189489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.189588 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.189692 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.189821 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260808 04:55:15.190016 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.180310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.15.30.125:44629)
0808 04:55:15.180651 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.180660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.180675 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:55:15.180709 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.180715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.180766 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.180854 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.181389 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.182095 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.183007 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.183259 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.183370 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.185601 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.185608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.185610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.185638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.188241 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.188754 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.188760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.188765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.188859 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.189219 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.189222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.189224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.189387 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.189492 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.189759 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.189861 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":153,"threads_started":1}
I20260808 04:55:15.190346 18693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:15.191095 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.190879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35024)
0808 04:55:15.190960 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.191019 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 04:55:15.191226 18581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } has no permanent_uuid. Determining permanent_uuid...
W20260808 04:55:15.191305 18617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
I20260808 04:55:15.191999 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.191674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35034)
0808 04:55:15.191855 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.191912 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 04:55:15.192306 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.191899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35038)
0808 04:55:15.192158 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.192211 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
W20260808 04:55:15.192667 18693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:39437: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
I20260808 04:55:15.192874 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.192536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35054)
0808 04:55:15.192730 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.192794 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260808 04:55:15.193051 18581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:39437: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
I20260808 04:55:15.224251 18581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } attempt: 1
I20260808 04:55:15.225142 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.224797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35058)
0808 04:55:15.224972 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.225046 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:55:15.225967 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.225714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35060)
0808 04:55:15.225846 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.225903 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 04:55:15.226143 18581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:39437: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
I20260808 04:55:15.241927 18693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } attempt: 1
I20260808 04:55:15.242666 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.242401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35076)
0808 04:55:15.242544 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.242599 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 04:55:15.243372 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.243154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:35086)
0808 04:55:15.243265 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.243318 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 04:55:15.243498 18693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:39437: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
W20260808 04:55:15.300410 18702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:55:15.300729 18702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:15.300791 18702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:15.300865 18702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 04:55:15.301337 18581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } attempt: 2
I20260808 04:55:15.302233 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.301867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:52248)
0808 04:55:15.302059 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.302134 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 04:55:15.302990 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.302767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:52256)
0808 04:55:15.302878 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.302930 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 04:55:15.303184 18581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:39437: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
W20260808 04:55:15.305594 18702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:15.305645 18702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:15.305665 18702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:15.305716 18702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:15.305744 18702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:15.310904 18702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:33395,127.15.30.125:44629,127.15.30.124:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab
--principal=kudu/127.15.30.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:39437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.30.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:15.311657 18693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } attempt: 2
I20260808 04:55:15.311921 18702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:15.312428 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.312110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:52268)
0808 04:55:15.312256 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.312314 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 04:55:15.313077 18702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:55:15.313138 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.312912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:52274)
0808 04:55:15.313022 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.313079 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:55:15.313273 18693 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:39437: Network error: Client connection negotiation failed: client connection to 127.15.30.124:39437: connect: Connection refused (error 111)
W20260808 04:55:15.321482 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:15.321456 18714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:15.321518 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:15.321910 18702 server_base.cc:1034] running on GCE node
I20260808 04:55:15.322531 18702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:15.323565 18702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:15.324723 18702 hybrid_clock.cc:648] HybridClock initialized: now 1786164915324702 us; error 30 us; skew 500 ppm
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:15.327486 18702 init.cc:377] Logged in from keytab as kudu/127.15.30.124@KRBTEST.COM (short username kudu)
I20260808 04:55:15.328344 18702 webserver.cc:460] Webserver started at http://127.15.30.124:34195/ using document root <none> and password file <none>
I20260808 04:55:15.328920 18702 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:15.329013 18702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:15.329280 18702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:15.331224 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data/instance:
uuid: "99eda067bc984d57be480a7ce1a8acc6"
format_stamp: "Formatted at 2026-08-08 04:55:15 on dist-test-slave-zpvx"
server_key: "eca2ad3069c9165f20aaeb79ef4b1e8e"
server_key_iv: "cdd18c901a719a3b9d1fa1896c07e9a9"
server_key_version: "encryptionkey@0"
I20260808 04:55:15.331768 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/wal/instance:
uuid: "99eda067bc984d57be480a7ce1a8acc6"
format_stamp: "Formatted at 2026-08-08 04:55:15 on dist-test-slave-zpvx"
server_key: "eca2ad3069c9165f20aaeb79ef4b1e8e"
server_key_iv: "cdd18c901a719a3b9d1fa1896c07e9a9"
server_key_version: "encryptionkey@0"
I20260808 04:55:15.335577 18702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:55:15.338050 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:15.339234 18702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 04:55:15.339383 18702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/wal
uuid: "99eda067bc984d57be480a7ce1a8acc6"
format_stamp: "Formatted at 2026-08-08 04:55:15 on dist-test-slave-zpvx"
server_key: "eca2ad3069c9165f20aaeb79ef4b1e8e"
server_key_iv: "cdd18c901a719a3b9d1fa1896c07e9a9"
server_key_version: "encryptionkey@0"
I20260808 04:55:15.339494 18702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:15.355670 18702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:15.358872 18702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:15.359097 18702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:15.366111 18702 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.124:39437
I20260808 04:55:15.366379 18809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.124:39437 every 8 connection(s)
I20260808 04:55:15.367067 18702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
I20260808 04:55:15.369812 18811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:15.373178 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 18702
I20260808 04:55:15.373354 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786164620544947-15481-0/minicluster-data/master-2/wal/instance
I20260808 04:55:15.373637 15481 external_mini_cluster.cc:1428] Setting key c688871a43e33c750a80c153c56134a4
I20260808 04:55:15.373564 18811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
I20260808 04:55:15.386662 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.375540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.15.30.126:33395)
0808 04:55:15.375693 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.375698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.375965 (+ 267us) server_negotiation.cc:407] Connection header received
0808 04:55:15.376632 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.376636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.376700 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.376806 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.377748 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.378265 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.379310 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.379549 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.379629 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.382458 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.382461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.382463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.382487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.384339 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.384916 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.384919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.384921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.385015 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.385417 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.385420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.385422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.385675 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.385750 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.386341 (+ 591us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.386494 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:55:15.387480 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.375454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:33395 (local address 127.0.0.1:53246)
0808 04:55:15.375836 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.375873 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.376413 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.376887 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.376893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.377296 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.377585 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.377608 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.378377 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.378381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.379181 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.379190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.379696 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.379700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.379868 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.380730 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.380746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.382268 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.384466 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.384474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.384481 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.384815 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.385134 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.385137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.385139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.385309 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.385798 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.385808 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.386068 (+ 260us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.386264 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.386606 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":66,"spinlock_wait_cycles":1920,"thread_start_us":138,"threads_started":1}
I20260808 04:55:15.387480 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.375096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.15.30.124:39437)
0808 04:55:15.375506 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.375524 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.375566 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:55:15.376409 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.376439 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.376630 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.376801 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.377548 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.378512 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.379366 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.379797 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.379964 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.382967 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.382974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.382986 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.383014 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.384908 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.385422 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.385428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.385430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.385507 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.385874 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.385876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.385878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.386126 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.386341 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.386576 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.386717 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260808 04:55:15.388841 18811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:55:15.397955 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.389466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.15.30.125:44629)
0808 04:55:15.389625 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.389628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.389641 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:55:15.389702 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.389705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.389759 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.389862 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.390451 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.391070 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.391986 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.392227 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.392305 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.394409 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.394412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.394414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.394436 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.396145 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.396717 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.396721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.396723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.396818 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.397109 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.397115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.397117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.397317 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.397425 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.397701 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.397819 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:55:15.397949 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.389369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:44629 (local address 127.0.0.1:54082)
0808 04:55:15.389517 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.389532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.389617 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.390001 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.390004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.390200 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.390326 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.390333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.391194 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.391197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.391865 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.391876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.392346 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.392352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.392481 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.392932 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.392947 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.394231 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.396407 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.396411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.396413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.396615 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.396926 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.396928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.396929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.397010 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.397420 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.397423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.397523 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.397636 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.397776 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 04:55:15.398687 18811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:15.398957 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.389824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.15.30.126:33395)
0808 04:55:15.389930 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.389934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.389955 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:55:15.390079 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.390082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.390121 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.390184 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.390859 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.391451 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.392422 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.392691 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.392804 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.395381 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.395385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.395387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.395417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.397071 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.397469 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.397475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.397476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.397571 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.397861 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.397864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.397866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.398309 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.398529 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.398670 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.398762 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:55:15.402344 18581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } attempt: 3
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
I20260808 04:55:15.404412 18693 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } attempt: 3
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
I20260808 04:55:15.407776 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.399208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:52300)
0808 04:55:15.399318 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.399332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.399404 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.399640 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.399643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.399806 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.399864 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.399869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.400796 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.400803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.401539 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.401550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.402036 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.402040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.402128 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.402581 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.402594 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.403900 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.406046 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.406049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.406050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.406288 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.406524 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.406526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.406528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.406589 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.407281 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.407284 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.407376 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.407469 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.407594 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 04:55:15.408339 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.399357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.15.30.124:39437)
0808 04:55:15.399461 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.399468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.399480 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:55:15.399506 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.399508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.399548 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.399629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.399972 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.400695 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.401693 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.401925 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.402037 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.404017 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.404023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.404026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.404055 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.405950 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.406382 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.406384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.406386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.406450 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.406941 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.406944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.406946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.407158 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.407248 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.408066 (+ 818us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.408189 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164914, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
I20260808 04:55:15.410735 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.400070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.15.30.125:44629)
0808 04:55:15.400165 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.400169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.400181 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:55:15.400302 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.400306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.400345 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.400459 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.401032 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.401580 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.402558 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.402867 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.402945 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.406352 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.406355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.406357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.406380 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.408460 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.408948 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.408952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.408954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.409060 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.409421 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.409426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.409427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.409849 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.410030 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.410446 (+ 416us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.410564 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Aug 08 04:55:15 dist-test-slave-zpvx krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164915, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
I20260808 04:55:15.412135 18811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: Bootstrap starting.
I20260808 04:55:15.413707 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.402726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:52308)
0808 04:55:15.403197 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.403211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.403286 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.403885 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.403888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.404087 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.404176 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.404181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.405858 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.405862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.406697 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.406710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.407411 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.407415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.407506 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.408088 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.408102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.409736 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.412068 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.412072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.412073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.412321 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.412682 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.412685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.412687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.412777 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.413215 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.413218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.413306 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.413386 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.413518 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395}
I20260808 04:55:15.414318 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.402911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.15.30.124:39437)
0808 04:55:15.403558 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.403568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.403599 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:55:15.403687 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.403691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.403739 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.403840 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.404296 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.405711 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.406849 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.407288 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.407706 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.409887 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.409891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.409893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.409916 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.411940 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.412478 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.412480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.412482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.412566 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.412890 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.412893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.412894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.413101 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.413955 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.414029 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.414155 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":558,"thread_start_us":130,"threads_started":1}
I20260808 04:55:15.414559 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:15.415185 18811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:15.416074 18811 log.cc:824] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:15.416970 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.404815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:39437 (local address 127.0.0.1:52314)
0808 04:55:15.405060 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:15.405075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:15.405160 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:15.405896 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:15.405899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:15.406116 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:15.406263 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.406270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.407571 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.407578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.408512 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:15.408524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.409197 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.409200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:15.409344 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.409853 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:15.409866 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:15.411298 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:15.415171 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.415174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.415177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.415436 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.415928 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:15.415931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:15.415933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:15.416029 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:15.416435 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:15.416438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:15.416539 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 04:55:15.416650 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 04:55:15.416788 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260808 04:55:15.416986 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.405301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.15.30.124:39437)
0808 04:55:15.405634 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.405644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.405657 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:55:15.405692 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.405698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.405746 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.406370 (+ 624us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.406509 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.407426 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.408663 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.409065 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.411717 (+ 2652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.411842 (+ 125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.411848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.411850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.411868 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.415028 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.415670 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.415674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.415675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.415811 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.416144 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.416146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.416147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.416297 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.416438 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.416716 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.416824 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
I20260808 04:55:15.417338 18581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72: Bootstrap starting.
I20260808 04:55:15.419308 18693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: Bootstrap starting.
I20260808 04:55:15.419461 18581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:15.420181 18581 log.cc:824] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:15.421746 18693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:15.422197 18581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72: No bootstrap required, opened a new log
I20260808 04:55:15.423095 18693 log.cc:824] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:15.420574 18530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1184e0cf1063406095ea18e52df8ee72"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53258
I20260808 04:55:15.425218 18811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: No bootstrap required, opened a new log
I20260808 04:55:15.425177 18581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
I20260808 04:55:15.425395 18581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1184e0cf1063406095ea18e52df8ee72, State: Initialized, Role: FOLLOWER
I20260808 04:55:15.425730 18693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: No bootstrap required, opened a new log
I20260808 04:55:15.425724 18581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
I20260808 04:55:15.426293 18827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.426491 18827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:15.427057 18581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:15.427124 18530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:55:15.427237 18530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:15.428519 18530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
I20260808 04:55:15.429546 18530 leader_election.cc:288] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [CANDIDATE]: Term 1 election: Requested vote from peers 59cd428d94f94b4c8a9955413b5a563a (127.15.30.125:44629), 99eda067bc984d57be480a7ce1a8acc6 (127.15.30.124:39437)
I20260808 04:55:15.430450 18693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
I20260808 04:55:15.430490 18811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
I20260808 04:55:15.430708 18693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cd428d94f94b4c8a9955413b5a563a, State: Initialized, Role: FOLLOWER
I20260808 04:55:15.430753 18811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99eda067bc984d57be480a7ce1a8acc6, State: Initialized, Role: FOLLOWER
I20260808 04:55:15.431075 18811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
I20260808 04:55:15.432392 18811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:15.432701 18693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
I20260808 04:55:15.433957 18693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:55:15.434301 18843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:15.434428 18843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:15.432101 18479 master_runner.cc:186] Error getting master registration for 127.15.30.125:44629: OK, instance_id { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" instance_seqno: 1786164915150243 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:55:15.436254 18648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1184e0cf1063406095ea18e52df8ee72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59cd428d94f94b4c8a9955413b5a563a"
I20260808 04:55:15.436511 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.436664 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:15.436674 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1184e0cf1063406095ea18e52df8ee72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99eda067bc984d57be480a7ce1a8acc6"
I20260808 04:55:15.436890 18770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:55:15.437863 18873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:15.437927 18873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:15.438337 18770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1184e0cf1063406095ea18e52df8ee72 in term 1.
I20260808 04:55:15.439464 18648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:15.440932 18648 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1184e0cf1063406095ea18e52df8ee72 in term 1.
I20260808 04:55:15.442458 18850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.442641 18850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:15.447216 18594 master_runner.cc:263] No leader master found from master 59cd428d94f94b4c8a9955413b5a563a
I20260808 04:55:15.447283 18594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:55:15.436087 18502 tablet.cc:2369] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:55:15.438848 18496 leader_election.cc:302] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1184e0cf1063406095ea18e52df8ee72, 99eda067bc984d57be480a7ce1a8acc6; no voters:
I20260808 04:55:15.448827 18479 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:55:15.449523 18827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:15.449856 18827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [term 1 LEADER]: Becoming Leader. State: Replica: 1184e0cf1063406095ea18e52df8ee72, State: Running, Role: LEADER
I20260808 04:55:15.451022 18702 master_runner.cc:263] No leader master found from master 99eda067bc984d57be480a7ce1a8acc6
I20260808 04:55:15.451085 18702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:55:15.453891 18895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 04:55:15.454862 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.437652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.15.30.126:33395)
0808 04:55:15.437766 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.437771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.440069 (+ 2298us) server_negotiation.cc:407] Connection header received
0808 04:55:15.440273 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.440276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.440320 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.440399 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.441576 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.442308 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.443522 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.443849 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.443929 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.450204 (+ 6275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.450207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.450210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.450236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.452834 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.453346 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.453352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.453355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.453450 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.453788 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.453792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.453793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.454035 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.454176 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.454555 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.454679 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:55:15.450443 18827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } }
W20260808 04:55:15.458235 18895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:15.460179 18827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1184e0cf1063406095ea18e52df8ee72. Latest consensus state: current_term: 1 leader_uuid: "1184e0cf1063406095ea18e52df8ee72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.460319 18827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:15.461954 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.438528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.15.30.125:44629)
0808 04:55:15.438637 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.438641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.444828 (+ 6187us) server_negotiation.cc:407] Connection header received
0808 04:55:15.445097 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.445100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.445146 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.445201 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.446195 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.446886 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.447990 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.448297 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.448402 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.456739 (+ 8337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.456743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.456746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.456772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.458543 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.460484 (+ 1941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.460490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.460492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.460603 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.460946 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.460952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.460957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.461202 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.461313 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.461687 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.461803 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:55:15.465835 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:15.439451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.15.30.124:39437)
0808 04:55:15.439563 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.439569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.454239 (+ 14670us) server_negotiation.cc:407] Connection header received
0808 04:55:15.454422 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.454428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.454489 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.454542 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.456059 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.456865 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.458444 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:15.458859 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:15.458949 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:15.461878 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:15.461882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:15.461884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:15.461910 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:15.463894 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.464423 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.464427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.464429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.464526 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:15.464856 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:15.464860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:15.464862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:15.465073 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:15.465186 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:15.465558 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 04:55:15.465695 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:55:15.466229 18905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:15.467360 18905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:15.476096 18770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [term 1 FOLLOWER]: Refusing update from remote peer 1184e0cf1063406095ea18e52df8ee72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:55:15.476884 18827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:55:15.477921 18648 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [term 1 FOLLOWER]: Refusing update from remote peer 1184e0cf1063406095ea18e52df8ee72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:55:15.480151 18827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:55:15.483912 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1184e0cf1063406095ea18e52df8ee72. Latest consensus state: current_term: 1 leader_uuid: "1184e0cf1063406095ea18e52df8ee72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.483912 18850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1184e0cf1063406095ea18e52df8ee72. Latest consensus state: current_term: 1 leader_uuid: "1184e0cf1063406095ea18e52df8ee72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.484025 18850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:15.484025 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:15.490844 18827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1184e0cf1063406095ea18e52df8ee72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.490963 18827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:15.491209 18850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1184e0cf1063406095ea18e52df8ee72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.491330 18850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:15.491314 18906 mvcc.cc:205] Tried to move back new op lower bound from 7316131493782192128 to 7316131493720281088. Current Snapshot: MvccSnapshot[applied={T|T < 7316131493782192128}]
I20260808 04:55:15.492480 18899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1184e0cf1063406095ea18e52df8ee72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.492591 18899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72 [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:15.493065 18908 mvcc.cc:205] Tried to move back new op lower bound from 7316131493782192128 to 7316131493720281088. Current Snapshot: MvccSnapshot[applied={T|T < 7316131493782192128}]
I20260808 04:55:15.493598 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1184e0cf1063406095ea18e52df8ee72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1184e0cf1063406095ea18e52df8ee72" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 33395 } } peers { permanent_uuid: "59cd428d94f94b4c8a9955413b5a563a" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 44629 } } peers { permanent_uuid: "99eda067bc984d57be480a7ce1a8acc6" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 39437 } } }
I20260808 04:55:15.493726 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:15.495357 18907 mvcc.cc:205] Tried to move back new op lower bound from 7316131493782192128 to 7316131493720281088. Current Snapshot: MvccSnapshot[applied={T|T < 7316131493782192128}]
I20260808 04:55:15.505684 18905 catalog_manager.cc:1347] Generated new cluster ID: 452bd7d044004cf2b46dcf1bc1d87fbd
I20260808 04:55:15.505753 18905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:15.530529 18905 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:15.532022 18905 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:15.547647 18905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1184e0cf1063406095ea18e52df8ee72: Generated new TSK 0
I20260808 04:55:15.548605 18905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:55:15.585325 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.582009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.15.30.125:44629)
0808 04:55:15.582143 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.582148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.584168 (+ 2020us) server_negotiation.cc:407] Connection header received
0808 04:55:15.584402 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.584406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.584458 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.584514 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.585180 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54122: BlockingRecv error: recv got EOF from 127.0.0.1:54122 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 04:55:15.587514 18587 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.582503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.15.30.126:33395)
0808 04:55:15.582630 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.582635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.585895 (+ 3260us) server_negotiation.cc:407] Connection header received
0808 04:55:15.586478 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.586481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.586532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.586593 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.587390 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53276: BlockingRecv error: recv got EOF from 127.0.0.1:53276 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 04:55:15.588441 18820 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.583311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.15.30.124:39437)
0808 04:55:15.583428 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.583433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.586216 (+ 2783us) server_negotiation.cc:407] Connection header received
0808 04:55:15.586729 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.586732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.586780 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.586830 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.588308 (+ 1478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52334: BlockingRecv error: recv got EOF from 127.0.0.1:52334 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 04:55:15.593139 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.591509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.15.30.125:44629)
0808 04:55:15.591621 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.591624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.592140 (+ 516us) server_negotiation.cc:407] Connection header received
0808 04:55:15.592351 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.592354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.592487 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.592542 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.593044 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54134: BlockingRecv error: recv got EOF from 127.0.0.1:54134 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 04:55:15.602638 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.601127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.15.30.125:44629)
0808 04:55:15.601231 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.601236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.601738 (+ 502us) server_negotiation.cc:407] Connection header received
0808 04:55:15.601920 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.601924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.601971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.602022 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.602527 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54150: BlockingRecv error: recv got EOF from 127.0.0.1:54150 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 04:55:15.608464 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.606859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.15.30.125:44629)
0808 04:55:15.606958 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.606962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.607556 (+ 594us) server_negotiation.cc:407] Connection header received
0808 04:55:15.607756 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.607759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.607806 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.607857 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.608352 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54154: BlockingRecv error: recv got EOF from 127.0.0.1:54154 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 04:55:15.616562 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.614900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.15.30.125:44629)
0808 04:55:15.615047 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.615051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.615549 (+ 498us) server_negotiation.cc:407] Connection header received
0808 04:55:15.615845 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.615848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.615899 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.615948 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.616468 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54160: BlockingRecv error: recv got EOF from 127.0.0.1:54160 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 04:55:15.628563 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.626914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.15.30.125:44629)
0808 04:55:15.627019 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.627023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.627546 (+ 523us) server_negotiation.cc:407] Connection header received
0808 04:55:15.627758 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.627761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.627810 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.627862 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.628464 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54168: BlockingRecv error: recv got EOF from 127.0.0.1:54168 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 04:55:15.649914 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.648031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.15.30.125:44629)
0808 04:55:15.648147 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.648152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.648759 (+ 607us) server_negotiation.cc:407] Connection header received
0808 04:55:15.648961 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.648964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.649042 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.649097 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.649773 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54174: BlockingRecv error: recv got EOF from 127.0.0.1:54174 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 04:55:15.686111 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.685002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.15.30.125:44629)
0808 04:55:15.685174 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.685178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.685250 (+ 72us) server_negotiation.cc:407] Connection header received
0808 04:55:15.685380 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.685383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.685431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.685507 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.685977 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54180: BlockingRecv error: recv got EOF from 127.0.0.1:54180 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 04:55:15.753726 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.752198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.15.30.125:44629)
0808 04:55:15.752349 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.752354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.752679 (+ 325us) server_negotiation.cc:407] Connection header received
0808 04:55:15.752768 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.752771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.752824 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.752903 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.753574 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54182: BlockingRecv error: recv got EOF from 127.0.0.1:54182 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:55:15.885571 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:15.884204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.15.30.125:44629)
0808 04:55:15.884426 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:15.884431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:15.884573 (+ 142us) server_negotiation.cc:407] Connection header received
0808 04:55:15.884717 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:15.884720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:15.884775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:15.884868 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:15.885390 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54190: BlockingRecv error: recv got EOF from 127.0.0.1:54190 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260808 04:55:16.145409 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:16.144095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.15.30.125:44629)
0808 04:55:16.144274 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:16.144279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:16.144457 (+ 178us) server_negotiation.cc:407] Connection header received
0808 04:55:16.144652 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:16.144655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:16.144714 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:16.144804 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:16.145275 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54206: BlockingRecv error: recv got EOF from 127.0.0.1:54206 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:55:16.439217 18873 catalog_manager.cc:1259] Loaded cluster ID: 452bd7d044004cf2b46dcf1bc1d87fbd
I20260808 04:55:16.439342 18873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: loading cluster ID for follower catalog manager: success
I20260808 04:55:16.441113 18873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: acquiring CA information for follower catalog manager: success
I20260808 04:55:16.442340 18873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99eda067bc984d57be480a7ce1a8acc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:55:16.459534 18895 catalog_manager.cc:1259] Loaded cluster ID: 452bd7d044004cf2b46dcf1bc1d87fbd
I20260808 04:55:16.459620 18895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: loading cluster ID for follower catalog manager: success
I20260808 04:55:16.461213 18895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: acquiring CA information for follower catalog manager: success
I20260808 04:55:16.462292 18895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59cd428d94f94b4c8a9955413b5a563a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:55:16.663712 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:16.659663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.15.30.125:44629)
0808 04:55:16.660010 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:16.660014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:16.660033 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:55:16.660127 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:16.660129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:16.660264 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:16.660395 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:16.660819 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.661396 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.662287 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.662518 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.662605 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:16.662895 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:16.663349 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:16.663446 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:16.663506 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 04:55:16.663580 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260808 04:55:16.670485 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:16.666908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.15.30.124:39437)
0808 04:55:16.667144 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:16.667148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:16.667163 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:55:16.667295 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:16.667297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:16.667410 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:16.667513 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:16.667704 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.668204 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.669106 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.669356 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.669447 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:16.669704 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:16.670131 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:16.670250 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:16.670311 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 04:55:16.670361 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260808 04:55:16.677651 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:16.673657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.15.30.125:44629)
0808 04:55:16.673780 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:16.673783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:16.674212 (+ 429us) server_negotiation.cc:407] Connection header received
0808 04:55:16.674363 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:16.674366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:16.674417 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:16.674502 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:16.674724 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.675500 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.676517 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.676736 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.676847 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:16.677111 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:16.677187 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:16.677272 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:16.677470 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 04:55:16.677526 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:55:16.678228 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:16.674053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.15.30.124:39437)
0808 04:55:16.674181 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:16.674186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:16.674202 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:55:16.674375 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:16.674378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:16.674428 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:16.674497 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:16.674900 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.675639 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.676889 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.677241 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.677323 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:16.677792 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:16.677881 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:16.677955 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:16.678060 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:55:16.678114 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:55:16.678661 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:16.673533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.15.30.126:33395)
0808 04:55:16.674511 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:16.674516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:16.674529 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:55:16.674627 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:16.674629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:16.674797 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:16.674952 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:16.675188 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.675833 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.676945 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:16.677261 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:16.677372 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:16.677709 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:16.678046 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:16.678413 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:16.678456 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 04:55:16.678517 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":77,"threads_started":1}
I20260808 04:55:16.685518 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18479
I20260808 04:55:16.700948 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18594
I20260808 04:55:16.711928 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 18702
2026-08-08T04:55:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3348 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:55:16 dist-test-slave-zpvx krb5kdc[19046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:55:16 dist-test-slave-zpvx krb5kdc[19046](info): set up 2 sockets
Aug 08 04:55:16 dist-test-slave-zpvx krb5kdc[19046](info): commencing operation
krb5kdc: starting...
W20260808 04:55:17.972736 15481 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/test-user.keytab.
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.30.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
Entry for principal kudu/127.15.30.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab.
I20260808 04:55:18.099541 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35401
--webserver_interface=127.15.30.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:35401,127.15.30.125:37697,127.15.30.124:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:55:18.212960 19066 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:55:18.213286 19066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:18.213349 19066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:18.213426 19066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:18.217450 19066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:18.217522 19066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:18.217561 19066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:18.217615 19066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:18.217653 19066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:18.222736 19066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:35401,127.15.30.125:37697,127.15.30.124:36763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.126.keytab
--principal=kudu/127.15.30.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.126:35401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.30.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:18.223706 19066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:18.224768 19066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:18.233346 19076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:18.233314 19077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:18.233319 19080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:18.233793 19066 server_base.cc:1034] running on GCE node
I20260808 04:55:18.234323 19066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:18.235404 19066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:18.236595 19066 hybrid_clock.cc:648] HybridClock initialized: now 1786164918236568 us; error 37 us; skew 500 ppm
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:18.239621 19066 init.cc:377] Logged in from keytab as kudu/127.15.30.126@KRBTEST.COM (short username kudu)
I20260808 04:55:18.240617 19066 webserver.cc:460] Webserver started at http://127.15.30.126:45377/ using document root <none> and password file <none>
I20260808 04:55:18.241173 19066 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:18.241267 19066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:18.241503 19066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:18.243517 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data/instance:
uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "39948a925fee18a19f263507be45fad9"
server_key_iv: "c64627f558652b234dfcbc110365d83c"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.244097 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/wal/instance:
uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "39948a925fee18a19f263507be45fad9"
server_key_iv: "c64627f558652b234dfcbc110365d83c"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.247874 19066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:55:18.250406 19093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:18.251793 19066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 04:55:18.251926 19066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/wal
uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "39948a925fee18a19f263507be45fad9"
server_key_iv: "c64627f558652b234dfcbc110365d83c"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.252028 19066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:18.277997 19066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:18.281145 19066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:18.281358 19066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:18.288446 19066 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.126:35401
I20260808 04:55:18.288723 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.126:35401 every 8 connection(s)
I20260808 04:55:18.289459 19066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/data/info.pb
I20260808 04:55:18.292155 19177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:18.295928 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 19066
I20260808 04:55:18.295783 19177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:18.296087 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-0/wal/instance
I20260808 04:55:18.296406 15481 external_mini_cluster.cc:1428] Setting key 13bea0b875c4328bb50c1f2d946fd0f3
WARNING: no policy specified for kudu/127.15.30.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.125@KRBTEST.COM" created.
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:55:18.313885 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.298145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35401 (local address 127.0.0.1:60976)
0808 04:55:18.298845 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.298897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.300120 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.300853 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.300858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.301309 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.301568 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.301588 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.302691 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.302694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.303407 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.303415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.304042 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.304046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.304167 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.304767 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.304783 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.308810 (+ 4027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.311002 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.311010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.311018 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.311238 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.311635 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.311637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.311639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.311740 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.312611 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.312616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.312744 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.312885 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.313140 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"spinlock_wait_cycles":6528,"thread_start_us":145,"threads_started":1}
I20260808 04:55:18.313894 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.298144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.15.30.126:35401)
0808 04:55:18.298550 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.298570 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.299020 (+ 450us) server_negotiation.cc:407] Connection header received
0808 04:55:18.300477 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.300481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.300669 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.300853 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.301715 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.302562 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.303535 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.303891 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.304056 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.308992 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.308999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.309009 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.309046 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.310870 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.311357 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.311367 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.311371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.311525 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.311836 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.311842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.311847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.312462 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.312628 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.313008 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.313258 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":262,"thread_start_us":129,"threads_started":1}
I20260808 04:55:18.316558 19177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:18.317235 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.316999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41164)
0808 04:55:18.317109 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.317186 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:55:18.317478 19097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
I20260808 04:55:18.318404 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.318143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41176)
0808 04:55:18.318276 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.318345 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 04:55:18.318651 19177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:37697: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab.
Entry for principal kudu/127.15.30.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab.
I20260808 04:55:18.325493 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:37697
--webserver_interface=127.15.30.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab
--principal=kudu/127.15.30.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:35401,127.15.30.125:37697,127.15.30.124:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:55:18.367890 19177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } attempt: 1
I20260808 04:55:18.368873 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.368537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41192)
0808 04:55:18.368720 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.368788 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 04:55:18.369746 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.369475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41200)
0808 04:55:18.369633 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.369688 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 04:55:18.369882 19177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:37697: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
I20260808 04:55:18.438042 19177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } attempt: 2
I20260808 04:55:18.438995 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.438693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41210)
0808 04:55:18.438852 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.438917 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:55:18.439914 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.439595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41218)
0808 04:55:18.439808 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.439861 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260808 04:55:18.440052 19177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.125:37697: Network error: Client connection negotiation failed: client connection to 127.15.30.125:37697: connect: Connection refused (error 111)
W20260808 04:55:18.442018 19195 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:55:18.442329 19195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:18.442395 19195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:18.442479 19195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:18.446802 19195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:18.446879 19195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:18.446926 19195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:18.446982 19195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:18.447019 19195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:18.452291 19195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:35401,127.15.30.125:37697,127.15.30.124:36763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.125.keytab
--principal=kudu/127.15.30.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.125:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.30.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:18.453323 19195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:18.454353 19195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:55:18.463166 19210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:18.463151 19204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:18.463287 19195 server_base.cc:1034] running on GCE node
W20260808 04:55:18.463151 19206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:18.464017 19195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:18.465186 19195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:18.466362 19195 hybrid_clock.cc:648] HybridClock initialized: now 1786164918466333 us; error 40 us; skew 500 ppm
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:18.469180 19195 init.cc:377] Logged in from keytab as kudu/127.15.30.125@KRBTEST.COM (short username kudu)
I20260808 04:55:18.470086 19195 webserver.cc:460] Webserver started at http://127.15.30.125:46349/ using document root <none> and password file <none>
I20260808 04:55:18.470664 19195 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:18.470727 19195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:18.470938 19195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:18.473068 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data/instance:
uuid: "d38b35577f6642db9ac36275cd3d1d7f"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "26ae94f4345fdd37586fc385c1719924"
server_key_iv: "88c21d040acef9f05dc79299c992cd20"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.473915 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/wal/instance:
uuid: "d38b35577f6642db9ac36275cd3d1d7f"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "26ae94f4345fdd37586fc385c1719924"
server_key_iv: "88c21d040acef9f05dc79299c992cd20"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.477767 19195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:55:18.480088 19219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:18.481395 19195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 04:55:18.481540 19195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/wal
uuid: "d38b35577f6642db9ac36275cd3d1d7f"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "26ae94f4345fdd37586fc385c1719924"
server_key_iv: "88c21d040acef9f05dc79299c992cd20"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.481657 19195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:18.499577 19195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:18.502302 19195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:18.502504 19195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:18.509944 19195 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.125:37697
I20260808 04:55:18.510233 19295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.125:37697 every 8 connection(s)
I20260808 04:55:18.510901 19195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/data/info.pb
I20260808 04:55:18.512243 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 19195
I20260808 04:55:18.512403 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-1/wal/instance
I20260808 04:55:18.512750 15481 external_mini_cluster.cc:1428] Setting key 0c84bede1e75f71d7245e9afeb5bb30e
I20260808 04:55:18.514317 19296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:18.518321 19296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.30.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.30.124@KRBTEST.COM" created.
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:55:18.531214 19177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } attempt: 3
I20260808 04:55:18.533325 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.520227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.15.30.126:35401)
0808 04:55:18.520405 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.520411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.521426 (+ 1015us) server_negotiation.cc:407] Connection header received
0808 04:55:18.522374 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.522377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.522442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.522548 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.523720 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.524407 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.526057 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.526295 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.526389 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.529186 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.529189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.529191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.529217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.530716 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.531356 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.531359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.531361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.531454 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.531907 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.531911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.531912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.532187 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.532270 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.532957 (+ 687us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.533165 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:55:18.534264 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.520757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35401 (local address 127.0.0.1:60984)
0808 04:55:18.521266 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.521308 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.522159 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.522733 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.522740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.523110 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.523564 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.523577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.524582 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.524586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.525909 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.525925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.526464 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.526471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.526741 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.527350 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.527366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.528985 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.530842 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.530849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.530867 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.531240 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.531592 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.531595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.531598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.531782 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.532354 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.532362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.532666 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.533091 (+ 425us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.533412 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
I20260808 04:55:18.536260 19296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.30.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab.
Entry for principal kudu/127.15.30.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab.
I20260808 04:55:18.543249 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.531686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41230)
0808 04:55:18.531847 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.531865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.531931 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.532858 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.532862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.533081 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.533216 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.533226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.534740 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.534747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.535531 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.535542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.536614 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.536620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.536815 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.537357 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.537371 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.538899 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.541387 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.541391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.541393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.541703 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.542047 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.542054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.542058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.542159 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.542724 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.542727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.542818 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.542889 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.543055 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:55:18.543439 15481 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdHhAno/build/debug/bin/kudu
/tmp/dist-test-taskdHhAno/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:36763
--webserver_interface=127.15.30.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab
--principal=kudu/127.15.30.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.30.84:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.30.126:35401,127.15.30.125:37697,127.15.30.124:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:55:18.543869 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.531990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.15.30.125:37697)
0808 04:55:18.532353 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.532388 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.532411 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:55:18.532447 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.532452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.532669 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.532859 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.533368 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.534459 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.535690 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.536487 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.536584 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.539039 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.539055 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.539065 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.539086 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.541270 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.541836 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.541842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.541844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.541933 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.542251 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.542256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.542259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.542585 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.543462 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.543589 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.543738 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":139,"threads_started":1}
I20260808 04:55:18.545814 19177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:18.546635 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.546327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33356)
0808 04:55:18.546502 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.546557 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 04:55:18.546900 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.536825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41234)
0808 04:55:18.536976 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.536993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.537079 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.537491 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.537494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.537660 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.537793 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.537806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.538857 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.538863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.539626 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.539638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.540208 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.540212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.540416 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.540838 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.540852 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.542463 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.545404 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.545408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.545411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.545663 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.545936 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.545939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.545940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.546017 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.546400 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.546405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.546490 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.546598 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.546730 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":22}
I20260808 04:55:18.547459 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.547189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33370)
0808 04:55:18.547369 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.547409 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260808 04:55:18.547586 19177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:36763: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
I20260808 04:55:18.546940 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.536936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.15.30.125:37697)
0808 04:55:18.537305 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.537312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.537323 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:55:18.537349 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.537350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.537383 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.537481 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.537907 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.538753 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.539757 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.540097 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.540208 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.542623 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.542627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.542630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.542659 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.545268 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.545757 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.545761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.545762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.545843 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.546106 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.546109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.546110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.546283 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.546390 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.546644 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.546784 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":307,"thread_start_us":133,"threads_started":1}
I20260808 04:55:18.548225 19296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:18.548955 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.548690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33374)
0808 04:55:18.548791 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.548875 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:55:18.549154 19222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
I20260808 04:55:18.550050 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.549778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33390)
0808 04:55:18.549907 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.549971 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:55:18.550215 19296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:36763: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
I20260808 04:55:18.578751 19177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } attempt: 1
I20260808 04:55:18.579545 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.579257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33404)
0808 04:55:18.579393 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.579463 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 04:55:18.580318 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.580105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33418)
0808 04:55:18.580214 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.580267 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:55:18.580462 19177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:36763: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
I20260808 04:55:18.599493 19296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } attempt: 1
I20260808 04:55:18.600263 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.599960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33424)
0808 04:55:18.600110 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.600184 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:55:18.600988 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.600769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33432)
0808 04:55:18.600875 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.600926 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 04:55:18.601168 19296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:36763: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
I20260808 04:55:18.655649 19177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } attempt: 2
I20260808 04:55:18.656579 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.656199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33448)
0808 04:55:18.656363 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.656465 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:55:18.657426 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.657204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33460)
0808 04:55:18.657311 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.657364 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 04:55:18.657585 19177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:36763: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
W20260808 04:55:18.657603 19310 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:55:18.657873 19310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:55:18.657895 19310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:55:18.657932 19310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:55:18.661821 19310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:55:18.661868 19310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:55:18.661888 19310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:55:18.661906 19310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:55:18.661921 19310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:55:18.666971 19310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.30.84:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.30.126:35401,127.15.30.125:37697,127.15.30.124:36763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/krb5kdc/kudu_127.15.30.124.keytab
--principal=kudu/127.15.30.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.30.124:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.30.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 04:05:18 UTC on ccycloud.quasar-drfdgt.root.comops.site
build id 4574101
I20260808 04:55:18.667879 19310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:55:18.668881 19310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:55:18.669330 19296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } attempt: 2
I20260808 04:55:18.670117 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.669821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33476)
0808 04:55:18.669971 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.670035 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:55:18.670902 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.670683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33492)
0808 04:55:18.670791 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.670841 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:55:18.671078 19296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.30.124:36763: Network error: Client connection negotiation failed: client connection to 127.15.30.124:36763: connect: Connection refused (error 111)
W20260808 04:55:18.677645 19324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:18.677682 19319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:55:18.677683 19326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:55:18.678156 19310 server_base.cc:1034] running on GCE node
I20260808 04:55:18.678797 19310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:55:18.679920 19310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:55:18.681092 19310 hybrid_clock.cc:648] HybridClock initialized: now 1786164918681055 us; error 45 us; skew 500 ppm
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:55:18.683991 19310 init.cc:377] Logged in from keytab as kudu/127.15.30.124@KRBTEST.COM (short username kudu)
I20260808 04:55:18.685112 19310 webserver.cc:460] Webserver started at http://127.15.30.124:35943/ using document root <none> and password file <none>
I20260808 04:55:18.685900 19310 fs_manager.cc:359] Metadata directory not provided
I20260808 04:55:18.686005 19310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:55:18.686262 19310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:55:18.688264 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data/instance:
uuid: "5bcf667dc1084d74bee6306ae62c3daa"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "6cee24bf30f38df16cf6707650e153d6"
server_key_iv: "3ff687ee70e171c8fd06a18cefe18ac9"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.688912 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/wal/instance:
uuid: "5bcf667dc1084d74bee6306ae62c3daa"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "6cee24bf30f38df16cf6707650e153d6"
server_key_iv: "3ff687ee70e171c8fd06a18cefe18ac9"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.693611 19310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:55:18.696499 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:55:18.697751 19310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 04:55:18.697901 19310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data,/tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/wal
uuid: "5bcf667dc1084d74bee6306ae62c3daa"
format_stamp: "Formatted at 2026-08-08 04:55:18 on dist-test-slave-zpvx"
server_key: "6cee24bf30f38df16cf6707650e153d6"
server_key_iv: "3ff687ee70e171c8fd06a18cefe18ac9"
server_key_version: "encryptionkey@0"
I20260808 04:55:18.698017 19310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:55:18.718425 19310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:55:18.721623 19310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:55:18.721815 19310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:55:18.728433 19310 rpc_server.cc:307] RPC server started. Bound to: 127.15.30.124:36763
I20260808 04:55:18.728739 19428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.30.124:36763 every 8 connection(s)
I20260808 04:55:18.729419 19310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/data/info.pb
I20260808 04:55:18.731338 15481 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdHhAno/build/debug/bin/kudu as pid 19310
I20260808 04:55:18.731475 15481 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdHhAno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786164620544947-15481-0/minicluster-data/master-2/wal/instance
I20260808 04:55:18.731822 15481 external_mini_cluster.cc:1428] Setting key 46c40e951ad9a7db46dc5a5c7acb79fc
I20260808 04:55:18.732544 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:55:18.737735 19430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
I20260808 04:55:18.746804 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.733247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.15.30.124:36763)
0808 04:55:18.733645 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.733661 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.733697 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:55:18.734367 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.734385 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.734724 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.735001 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.735923 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.737040 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.738246 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.738885 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.739037 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.741937 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.741943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.741954 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.741984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.744047 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.744547 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.744553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.744559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.744699 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.744988 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.744994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.744999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.745383 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.745533 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.745831 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.746118 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260808 04:55:18.750283 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.739695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.15.30.126:35401)
0808 04:55:18.739852 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.739857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.740176 (+ 319us) server_negotiation.cc:407] Connection header received
0808 04:55:18.740348 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.740352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.740457 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.740539 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.741382 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.742142 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.743308 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.743656 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.743775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.746169 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.746172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.746174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.746197 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.748202 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.748780 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.748785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.748787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.748888 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.749246 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.749250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.749252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.749479 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.749558 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.750003 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.750128 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:55:18.750301 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.739627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.126:35401 (local address 127.0.0.1:60994)
0808 04:55:18.740040 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.740079 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.740228 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.740701 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.740708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.741108 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.741236 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.741242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.742319 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.742324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.743164 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.743174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.743780 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.743782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.743863 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.744321 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.744333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.746026 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.748317 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.748328 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.748333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.748668 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.749008 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.749012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.749013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.749123 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.749657 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.749665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.749784 (+ 119us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.749948 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.750121 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":137,"threads_started":1}
I20260808 04:55:18.751505 19430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.126@KRBTEST.COM
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
I20260808 04:55:18.756749 19177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } attempt: 3
I20260808 04:55:18.760321 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.749050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.15.30.126:35401)
0808 04:55:18.749517 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.749526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.749540 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:55:18.749573 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.749579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.749626 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.749713 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.750548 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.751288 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.752280 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.752642 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.752747 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.755336 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.755341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.755343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.755375 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.757659 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.758907 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.758914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.758916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.759016 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.759361 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.759367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.759370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.759550 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.759769 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.759925 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.760045 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":120,"threads_started":1}
I20260808 04:55:18.760490 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.752265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.15.30.125:37697)
0808 04:55:18.752441 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.752446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.752461 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:55:18.752510 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.752512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.752553 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.752633 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.753076 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.753582 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.754350 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.754604 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.754718 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.756805 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.756809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.756812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.756840 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.758486 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.758989 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.758995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.759000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.759092 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.759434 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.759438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.759440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.759677 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.759779 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.760137 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.760268 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:55:18.761144 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.752111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.125:37697 (local address 127.0.0.1:41240)
0808 04:55:18.752279 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.752292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.752355 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.752679 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.752684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.752858 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.752950 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.752955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.753692 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.753697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.754229 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.754238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.754718 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.754721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.754813 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.755259 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.755275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.756637 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.758599 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.758601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.758603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.758805 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.759203 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.759207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.759209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.759297 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.759829 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.759833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.759923 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.760771 (+ 848us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.760961 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 04:55:18.761904 19430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:55:18.762249 19296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } attempt: 3
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.126@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
I20260808 04:55:18.767743 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.757165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33506)
0808 04:55:18.757272 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.757289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.757372 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.757866 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.757868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.758007 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.758079 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.758094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.759059 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.759066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.759974 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.759987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.761541 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.761545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.761638 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.762094 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.762110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.763668 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.765668 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.765673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.765677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.765928 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.766490 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.766493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.766496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.766581 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.766992 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.766995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.767095 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.767175 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.767304 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.30.125@KRBTEST.COM
I20260808 04:55:18.768687 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.757428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.15.30.124:36763)
0808 04:55:18.757573 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.757578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.757592 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:55:18.757631 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.757634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.757675 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.757756 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.758215 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.758970 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.760137 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.761267 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.761387 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.763810 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.763812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.763814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.763832 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.765532 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.766310 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.766314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.766315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.766393 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.766704 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.766707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.766709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.766882 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.768294 (+ 1412us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.768399 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.768513 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.124@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
Aug 08 04:55:18 dist-test-slave-zpvx krb5kdc[19046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786164918, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.30.125@KRBTEST.COM for kudu/127.15.30.124@KRBTEST.COM
I20260808 04:55:18.772791 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.761561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.15.30.125:37697)
0808 04:55:18.761770 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.761777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.761797 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:55:18.762237 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.762241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.762296 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.762434 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.763176 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.763958 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.765267 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.765587 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.766040 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.768510 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.768514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.768516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.768541 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.770417 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.770946 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.770951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.770954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.771042 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.771343 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.771348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.771352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.771582 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.772335 (+ 753us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.772463 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.772612 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":103}
I20260808 04:55:18.773146 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.762382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33522)
0808 04:55:18.763856 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.763870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.763947 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.764267 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.764273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.764504 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.764606 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.764616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.765754 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.765760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.766552 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.766560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.767123 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.767127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.767207 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.767619 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.767633 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.769272 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.771638 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.771641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.771643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.771901 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.772198 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.772204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.772209 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.772306 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.772740 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.772742 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.772814 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.772888 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.772999 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"spinlock_wait_cycles":7040}
I20260808 04:55:18.773759 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd: Bootstrap starting.
I20260808 04:55:18.774037 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.762630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.30.124:36763 (local address 127.0.0.1:33526)
0808 04:55:18.762817 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:55:18.762832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:55:18.762907 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:55:18.764457 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:55:18.764460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:55:18.764753 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:55:18.764847 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.764854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.766050 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.766053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.766976 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:55:18.766992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.767689 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.767694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:55:18.767786 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.768249 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:55:18.768267 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:55:18.769900 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:55:18.772493 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.772496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.772498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.772742 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.773058 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:55:18.773062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:55:18.773064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:55:18.773152 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:55:18.773530 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:55:18.773534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:55:18.773638 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 04:55:18.773744 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:55:18.773878 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":3}
I20260808 04:55:18.774037 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.762557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.15.30.124:36763)
0808 04:55:18.763459 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.763466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.763935 (+ 469us) server_negotiation.cc:407] Connection header received
0808 04:55:18.764042 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.764046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.764094 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.764717 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.764819 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.765656 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.766676 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.767010 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.767396 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.769411 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.769414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.769415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.769458 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.771530 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.771998 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.772001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.772003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.772086 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.772435 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.772438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.772439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.772639 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.773632 (+ 993us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.773703 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.773812 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":107,"threads_started":1}
I20260808 04:55:18.774104 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.763138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.15.30.124:36763)
0808 04:55:18.764172 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.764181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.764193 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:55:18.764227 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.764230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.764271 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.764964 (+ 693us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.765146 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.765919 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.767159 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.767537 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.767994 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.770055 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.770058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.770061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.770088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.772314 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.772863 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.772866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.772868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.772948 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.773248 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.773253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.773255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.773442 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.773534 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.773817 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.773918 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"spinlock_wait_cycles":2432,"thread_start_us":120,"threads_started":1}
I20260808 04:55:18.777030 19177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:18.777324 19296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: Bootstrap starting.
I20260808 04:55:18.777324 19430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: Bootstrap starting.
I20260808 04:55:18.777953 19177 log.cc:824] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:18.778157 15481 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:55:18.779723 19430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:18.779723 19296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: Neither blocks nor log segments found. Creating new log.
I20260808 04:55:18.780079 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd: No bootstrap required, opened a new log
I20260808 04:55:18.780572 19430 log.cc:824] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:18.780572 19296 log.cc:824] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:55:18.779364 19138 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32770
I20260808 04:55:18.782562 19430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: No bootstrap required, opened a new log
I20260808 04:55:18.782810 19296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: No bootstrap required, opened a new log
I20260808 04:55:18.783143 19177 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.783344 19177 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ed5ae2f4ed4c16b5c7cab38c8625dd, State: Initialized, Role: FOLLOWER
I20260808 04:55:18.783671 19177 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.784179 19445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.784448 19445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:18.784970 19177 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:18.785001 19138 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:55:18.785118 19138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:18.786375 19138 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.786774 19296 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.786994 19296 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38b35577f6642db9ac36275cd3d1d7f, State: Initialized, Role: FOLLOWER
I20260808 04:55:18.787218 19430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.787369 19138 leader_election.cc:288] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [CANDIDATE]: Term 1 election: Requested vote from peers d38b35577f6642db9ac36275cd3d1d7f (127.15.30.125:37697), 5bcf667dc1084d74bee6306ae62c3daa (127.15.30.124:36763)
I20260808 04:55:18.787429 19430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcf667dc1084d74bee6306ae62c3daa, State: Initialized, Role: FOLLOWER
I20260808 04:55:18.787284 19296 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.787801 19430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.788816 19296 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:18.788926 19430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:55:18.790251 19383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5bcf667dc1084d74bee6306ae62c3daa"
I20260808 04:55:18.790473 19383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:18.791715 19383 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ed5ae2f4ed4c16b5c7cab38c8625dd in term 1.
I20260808 04:55:18.792124 19259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d38b35577f6642db9ac36275cd3d1d7f"
I20260808 04:55:18.792344 19259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:55:18.793583 19259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ed5ae2f4ed4c16b5c7cab38c8625dd in term 1.
I20260808 04:55:18.790463 19066 master_runner.cc:186] Error getting master registration for 127.15.30.125:37697: OK, instance_id { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" instance_seqno: 1786164918507852 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:55:18.793962 19097 leader_election.cc:302] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ed5ae2f4ed4c16b5c7cab38c8625dd, d38b35577f6642db9ac36275cd3d1d7f; no voters:
I20260808 04:55:18.794557 19465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
W20260808 04:55:18.794799 19459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:18.794801 19507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:18.794879 19507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 04:55:18.794879 19459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:18.795281 19465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:18.798758 19066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:55:18.798866 19445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:55:18.800794 19195 master_runner.cc:263] No leader master found from master d38b35577f6642db9ac36275cd3d1d7f
I20260808 04:55:18.800884 19195 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:55:18.800791 19462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.801028 19462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:18.801578 19310 master_runner.cc:263] No leader master found from master 5bcf667dc1084d74bee6306ae62c3daa
I20260808 04:55:18.801641 19310 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:55:18.802855 19515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:55:18.802937 19515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:55:18.803143 19445 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [term 1 LEADER]: Becoming Leader. State: Replica: a9ed5ae2f4ed4c16b5c7cab38c8625dd, State: Running, Role: LEADER
I20260808 04:55:18.803457 19445 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } }
I20260808 04:55:18.805188 19445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ed5ae2f4ed4c16b5c7cab38c8625dd. Latest consensus state: current_term: 1 leader_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.805300 19445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:18.811239 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.795876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.15.30.126:35401)
0808 04:55:18.795982 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.795986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.799275 (+ 3289us) server_negotiation.cc:407] Connection header received
0808 04:55:18.799466 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.799469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.799522 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.799573 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.800865 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.803765 (+ 2900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.804983 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.805331 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.805429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.807241 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.807245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.807248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.807274 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.809143 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.809654 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.809659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.809662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.809763 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.810147 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.810153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.810157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.810408 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.810548 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.810931 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.811055 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 04:55:18.813469 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.796102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.15.30.124:36763)
0808 04:55:18.796465 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.796469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.801120 (+ 4651us) server_negotiation.cc:407] Connection header received
0808 04:55:18.801297 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.801301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.801344 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.801399 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.802253 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.802889 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.807015 (+ 4126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.807333 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.807410 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.809341 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.809348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.809353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.809380 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.811145 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.811856 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.811859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.811861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.811950 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.812632 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.812636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.812638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.812811 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.812893 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.813191 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.813300 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260808 04:55:18.814862 19528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:55:18.815835 19528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:55:18.818925 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:18.796696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.15.30.125:37697)
0808 04:55:18.796805 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.796809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.802890 (+ 6081us) server_negotiation.cc:407] Connection header received
0808 04:55:18.803044 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.803048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.803094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.803146 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.811463 (+ 8317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.812231 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.813408 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:18.813743 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:18.813818 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:18.815648 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:55:18.815651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:55:18.815653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:55:18.815676 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:55:18.817144 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.817574 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.817578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.817579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.817658 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:55:18.817979 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:55:18.817982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:55:18.817984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:55:18.818194 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:55:18.818317 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:18.818676 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 04:55:18.818789 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:55:18.821406 19383 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [term 1 FOLLOWER]: Refusing update from remote peer a9ed5ae2f4ed4c16b5c7cab38c8625dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:55:18.822168 19259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [term 1 FOLLOWER]: Refusing update from remote peer a9ed5ae2f4ed4c16b5c7cab38c8625dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:55:18.822813 19445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:55:18.823211 19445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:55:18.827498 19465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ed5ae2f4ed4c16b5c7cab38c8625dd. Latest consensus state: current_term: 1 leader_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.827641 19465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:18.830937 19462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ed5ae2f4ed4c16b5c7cab38c8625dd. Latest consensus state: current_term: 1 leader_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.831055 19462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:18.834244 19525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.834378 19525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:18.834817 19530 mvcc.cc:205] Tried to move back new op lower bound from 7316131507485691904 to 7316131507422224384. Current Snapshot: MvccSnapshot[applied={T|T < 7316131507485691904}]
I20260808 04:55:18.837306 19445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.837419 19445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd [sys.catalog]: This master's current role is: LEADER
I20260808 04:55:18.838433 19462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.838557 19462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:18.839798 19465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ed5ae2f4ed4c16b5c7cab38c8625dd" member_type: VOTER last_known_addr { host: "127.15.30.126" port: 35401 } } peers { permanent_uuid: "d38b35577f6642db9ac36275cd3d1d7f" member_type: VOTER last_known_addr { host: "127.15.30.125" port: 37697 } } peers { permanent_uuid: "5bcf667dc1084d74bee6306ae62c3daa" member_type: VOTER last_known_addr { host: "127.15.30.124" port: 36763 } } }
I20260808 04:55:18.839910 19465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:55:18.840555 19531 mvcc.cc:205] Tried to move back new op lower bound from 7316131507485691904 to 7316131507422224384. Current Snapshot: MvccSnapshot[applied={T|T < 7316131507485691904}]
I20260808 04:55:18.842134 19528 catalog_manager.cc:1347] Generated new cluster ID: be25d80ef259441f80fa4c97283a30cd
I20260808 04:55:18.842198 19528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:55:18.842775 19537 mvcc.cc:205] Tried to move back new op lower bound from 7316131507485691904 to 7316131507422224384. Current Snapshot: MvccSnapshot[applied={T|T < 7316131507485691904}]
I20260808 04:55:18.912731 19528 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:55:18.914050 19528 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:55:18.923739 19528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9ed5ae2f4ed4c16b5c7cab38c8625dd: Generated new TSK 0
I20260808 04:55:18.924587 19528 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:55:18.955428 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.952401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.15.30.125:37697)
0808 04:55:18.952529 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.952533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.954321 (+ 1788us) server_negotiation.cc:407] Connection header received
0808 04:55:18.954516 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.954519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.954571 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.954639 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.955301 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41268: BlockingRecv error: recv got EOF from 127.0.0.1:41268 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 04:55:18.956878 19434 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.952854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.15.30.124:36763)
0808 04:55:18.952998 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.953003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.955951 (+ 2948us) server_negotiation.cc:407] Connection header received
0808 04:55:18.956058 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.956061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.956126 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.956215 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.956756 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33552: BlockingRecv error: recv got EOF from 127.0.0.1:33552 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 04:55:18.957199 19436 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.953187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.15.30.126:35401)
0808 04:55:18.953305 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.953310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.956225 (+ 2915us) server_negotiation.cc:407] Connection header received
0808 04:55:18.956425 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.956429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.956476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.956528 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.957088 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32788: BlockingRecv error: recv got EOF from 127.0.0.1:32788 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 04:55:18.961524 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.960020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.15.30.125:37697)
0808 04:55:18.960118 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.960122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.960648 (+ 526us) server_negotiation.cc:407] Connection header received
0808 04:55:18.960855 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.960858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.960926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.960976 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.961434 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41274: BlockingRecv error: recv got EOF from 127.0.0.1:41274 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 04:55:18.966509 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.965075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.15.30.125:37697)
0808 04:55:18.965168 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.965172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.965682 (+ 510us) server_negotiation.cc:407] Connection header received
0808 04:55:18.965868 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.965871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.965918 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.965963 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.966412 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41282: BlockingRecv error: recv got EOF from 127.0.0.1:41282 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 04:55:18.972057 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.970575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.15.30.125:37697)
0808 04:55:18.970679 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.970683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.971219 (+ 536us) server_negotiation.cc:407] Connection header received
0808 04:55:18.971400 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.971402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.971447 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.971495 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.971969 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41290: BlockingRecv error: recv got EOF from 127.0.0.1:41290 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 04:55:18.979557 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.978093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.15.30.125:37697)
0808 04:55:18.978220 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.978226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.978712 (+ 486us) server_negotiation.cc:407] Connection header received
0808 04:55:18.978888 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.978892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.978938 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.978986 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.979471 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41306: BlockingRecv error: recv got EOF from 127.0.0.1:41306 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 04:55:18.991724 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:18.990004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.15.30.125:37697)
0808 04:55:18.990178 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:18.990184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:18.990764 (+ 580us) server_negotiation.cc:407] Connection header received
0808 04:55:18.990973 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:18.990977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:18.991031 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:18.991082 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:18.991616 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41314: BlockingRecv error: recv got EOF from 127.0.0.1:41314 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 04:55:19.011937 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:19.010249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.15.30.125:37697)
0808 04:55:19.010365 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:19.010369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:19.010949 (+ 580us) server_negotiation.cc:407] Connection header received
0808 04:55:19.011144 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:19.011148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:19.011202 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:19.011254 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:19.011820 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41320: BlockingRecv error: recv got EOF from 127.0.0.1:41320 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 04:55:19.047614 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:19.046413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.15.30.125:37697)
0808 04:55:19.046621 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:19.046625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:19.046704 (+ 79us) server_negotiation.cc:407] Connection header received
0808 04:55:19.046824 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:19.046827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:19.046877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:19.046952 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:19.047464 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41324: BlockingRecv error: recv got EOF from 127.0.0.1:41324 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260808 04:55:19.115121 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:19.113862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.15.30.125:37697)
0808 04:55:19.114048 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:19.114051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:19.114199 (+ 148us) server_negotiation.cc:407] Connection header received
0808 04:55:19.114271 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:19.114274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:19.114335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:19.114421 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:19.115008 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41326: BlockingRecv error: recv got EOF from 127.0.0.1:41326 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 04:55:19.246740 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:19.245441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.15.30.125:37697)
0808 04:55:19.245604 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:19.245607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:19.245746 (+ 139us) server_negotiation.cc:407] Connection header received
0808 04:55:19.245950 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:19.245953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:19.246009 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:19.246139 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:19.246639 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41334: BlockingRecv error: recv got EOF from 127.0.0.1:41334 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 04:55:19.506698 19309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:55:19.505428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.15.30.125:37697)
0808 04:55:19.505593 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:19.505597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:19.505695 (+ 98us) server_negotiation.cc:407] Connection header received
0808 04:55:19.505878 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:19.505881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:19.505935 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:19.506023 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:55:19.506540 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41338: BlockingRecv error: recv got EOF from 127.0.0.1:41338 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:55:19.796258 19507 catalog_manager.cc:1259] Loaded cluster ID: be25d80ef259441f80fa4c97283a30cd
I20260808 04:55:19.796366 19507 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: loading cluster ID for follower catalog manager: success
I20260808 04:55:19.798060 19507 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: acquiring CA information for follower catalog manager: success
I20260808 04:55:19.799209 19507 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d38b35577f6642db9ac36275cd3d1d7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:55:19.804003 19515 catalog_manager.cc:1259] Loaded cluster ID: be25d80ef259441f80fa4c97283a30cd
I20260808 04:55:19.804075 19515 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: loading cluster ID for follower catalog manager: success
I20260808 04:55:19.805655 19515 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: acquiring CA information for follower catalog manager: success
I20260808 04:55:19.806772 19515 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5bcf667dc1084d74bee6306ae62c3daa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:55:20.025104 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:20.021129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.15.30.125:37697)
0808 04:55:20.021372 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:20.021376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:20.021414 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:55:20.021575 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:20.021578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:20.021717 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:20.021819 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:20.022150 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.022687 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.023551 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.023778 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.023860 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:20.024141 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:20.024654 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:20.024749 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:20.024824 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 04:55:20.024872 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260808 04:55:20.031667 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:20.028018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.15.30.124:36763)
0808 04:55:20.028251 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:20.028254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:20.028268 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:55:20.028469 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:20.028471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:20.028586 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:20.028689 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:20.028925 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.029491 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.030292 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.030525 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.030614 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:20.030920 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:20.031347 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:20.031445 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:20.031507 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 04:55:20.031553 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260808 04:55:20.039207 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:20.034995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.15.30.126:35401)
0808 04:55:20.035387 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:20.035391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:20.035404 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:55:20.035493 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:20.035496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:20.035642 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:20.035781 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:20.035996 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.036791 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.037625 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.037918 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.038018 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:20.038294 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:20.038572 (+ 278us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:20.038660 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:20.038996 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 04:55:20.039058 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":68,"threads_started":1}
I20260808 04:55:20.039207 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:20.035194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.15.30.124:36763)
0808 04:55:20.035312 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:20.035316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:20.035329 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:55:20.035370 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:20.035373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:20.035423 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:20.035495 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:20.035823 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.036611 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.037936 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.038260 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.038349 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:20.038706 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:20.038803 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:20.038872 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:20.038977 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:55:20.039036 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:55:20.039505 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:55:20.034663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.15.30.125:37697)
0808 04:55:20.035101 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 04:55:20.035105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:55:20.035340 (+ 235us) server_negotiation.cc:407] Connection header received
0808 04:55:20.035480 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:55:20.035483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:55:20.035536 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:55:20.035631 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:55:20.035896 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.036515 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.038107 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:55:20.038443 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:55:20.038540 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:55:20.038831 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:55:20.038918 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:55:20.039004 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:55:20.039325 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 04:55:20.039388 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260808 04:55:20.042415 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 19066
I20260808 04:55:20.060353 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 19195
I20260808 04:55:20.071909 15481 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdHhAno/build/debug/bin/kudu with pid 19310
2026-08-08T04:55:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3361 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6710 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299526 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST